BUG: f2py: Exception when trying to parse private/public subroutine declaration in combination with only: option

[m-weigand]

Describe the issue:

When trying to build a Python module for a legacy Fortran 90 code base, I think I came across a small bug in f2py.

When f2py is used with the only: directive to only selectively map routines, then private/public statements for subroutines in f90 modules will be mistaken as variables (the subroutines themselves will be filtered out at an earlier step). This leads to an exception in f2py because no type for those 'variables' can be inferred. In the example below, the module mod2 has two public subroutines, which will be filtered by a corresponding only: option for f2py. yet, the definitions

    module mod2
        implicit none
        PUBLIC :: hey_do
        PUBLIC :: do_hey
    contains

will remain, leading to the problems described here.

My interpretation of this is: f2py will remove the body of subroutines that are not listed in the onlyfuncsvariable (in crackfortran.py), which then leads to the remaining artifacts in the declaration of the module being mistaken as variables (f90mod_rules.py).

I was able to work around this issue by adding a check into the corresponding function, which I believe filters out those remaining public/private declarations (while I'm not deeply familiar with Fortran 90 syntax, I believe that anything in the form PUBLIC :: [NAME] or PRIVATE :: [NAME]: must always refer to subroutines. Variables should always require a type?).

if len(var) == 1 and 'attrspec' in var and var['attrspec'][0] in ('public', 'private'):

This, however, lead to another issue: It is now possible that a module has no attributes or subroutines to export (either because everything is private, or filtered out due to only:). This leads to the generation of boilerplate C code for the empty module that crashes Python when the generated module is loaded. To solve this, I added another check that skips a given module if neither variables or anything else is to be exported.
The second issue can be prevented by either USEin the module, or by adding public dummy variables.

The work-around patch for current numpy HEAD (abeca76):

diff --git a/numpy/f2py/f90mod_rules.py b/numpy/f2py/f90mod_rules.py
index db53beaf61..5b4dcf2ff6 100644
--- a/numpy/f2py/f90mod_rules.py
+++ b/numpy/f2py/f90mod_rules.py
@@ -110,11 +110,20 @@ def dadd(line, s=doc):
                 notvars.append(b['name'])
         for n in m['vars'].keys():
             var = m['vars'][n]
-            if (n not in notvars) and (not l_or(isintent_hide, isprivate)(var)):
+            if len(var) == 1 and 'attrspec' in var and var['attrspec'][0] in ('public', 'private'):
+                is_not_var = True
+            else:
+                is_not_var = False
+
+            if (n not in notvars and not is_not_var) and (not l_or(isintent_hide, isprivate)(var)):
                 onlyvars.append(n)
                 mfargs.append(n)
         outmess('\t\tConstructing F90 module support for "%s"...\n' %
                 (m['name']))
+        if len(onlyvars) == 0 and len(notvars) == 1 and m['name'] in notvars:
+            outmess(f"\t\t\tSkipping {m['name']} since there are not vars/func...\n")
+            continue
+
         if m['name'] in usenames and not contains_functions_or_subroutines:
             outmess(f"\t\t\tSkipping {m['name']} since it is in 'use'...\n")
             continue

Reproduce the code example:

$ cat tmod.f90

    module mod2
        implicit none
        PUBLIC :: hey_do
        PUBLIC :: do_hey
    contains

        subroutine hey_do()
            implicit none
        end subroutine hey_do

        subroutine do_hey()
            implicit none
        end subroutine do_hey
    end module mod2

    module mod1
        implicit none
        private
        PUBLIC :: func1
    contains

        subroutine func1()
        end subroutine func1

    end module mod1

$ f2py -c --backend=meson --build-dir output tmod.f90 only: func1

The Fortran code above compiles cleanly with gfortran:

    (numpy) mweigand@c5c53cfebebc:~/t1$ cat main.f90
    program main
        use tmod
    end program main

    (numpy) mweigand@c5c53cfebebc:~/t1$  gfortran -Wall tmod.f90 main.f90

Error message:

(numpy) mweigand@c5c53cfebebc:~/t1$ f2py -c --backend=meson --build-dir output tmod.f90 only: func1
    Using meson backend
    Will pass --lower to f2py
    See https://numpy.org/doc/stable/f2py/buildtools/meson.html
    Reading fortran codes...
            Reading file 'tmod.f90' (format:free)
    Post-processing...
            Block: untitled
                            Block: mod2
                            Block: mod1
                                    Block: func1
    Applying post-processing hooks...
      character_backward_compatibility_hook
    Post-processing (stage 2)...
            Block: untitled
                    Block: unknown_interface
                            Block: mod2
                            Block: mod1
                                    Block: func1
    Building modules...
        Building module "untitled"...
                    Constructing F90 module support for "mod2"...
                      Variables: hey_do do_hey
    getctype: No C-type found in "{'attrspec': ['public']}", assuming void.
    Traceback (most recent call last):
      File "/home/mweigand/.virtualenvs/numpy/bin/f2py", line 8, in <module>
        sys.exit(main())
                 ^^^^^^
      File "/home/mweigand/.virtualenvs/numpy/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 765, in main
        run_compile()
      File "/home/mweigand/.virtualenvs/numpy/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 711, in run_compile
        run_main(f" {' '.join(f2py_flags)} -m {modulename} {' '.join(sources)}".split())
      File "/home/mweigand/.virtualenvs/numpy/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 496, in run_main
        ret = buildmodules(postlist)
              ^^^^^^^^^^^^^^^^^^^^^^
      File "/home/mweigand/.virtualenvs/numpy/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 401, in buildmodules
        dict_append(ret[name], rules.buildmodule(module, um))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/mweigand/.virtualenvs/numpy/lib/python3.12/site-packages/numpy/f2py/rules.py", line 1312, in buildmodule
        mr, wrap = f90mod_rules.buildhooks(m)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/mweigand/.virtualenvs/numpy/lib/python3.12/site-packages/numpy/f2py/f90mod_rules.py", line 146, in buildhooks
        at = capi_maps.c2capi_map[ct]
             ~~~~~~~~~~~~~~~~~~~~^^^^
    KeyError: 'void'

Python and NumPy Versions:

python -c "import sys, numpy; print(numpy.__version__); print(sys.version)"
2.0.0
3.12.4 (main, Jun 12 2024, 19:06:53) [GCC 13.2.0]

Runtime Environment:

import numpy; numpy.show_runtime()
[{'numpy_version': '2.0.0',
'python': '3.12.4 (main, Jun 12 2024, 19:06:53) [GCC 13.2.0]',
'uname': uname_result(system='Linux', node='c5c53cfebebc', release='6.1.0-22-amd64', version='#1 SMP PREEMPT_DYNAMIC Debian 6.1.94-1 (2024-06-21)', machine='x86_64')},
{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
'found': ['SSSE3',
'SSE41',
'POPCNT',
'SSE42',
'AVX',
'F16C',
'FMA3',
'AVX2'],
'not_found': ['AVX512F',
'AVX512CD',
'AVX512_KNL',
'AVX512_KNM',
'AVX512_SKX',
'AVX512_CLX',
'AVX512_CNL',
'AVX512_ICL']}},
{'architecture': 'Haswell',
'filepath': '/home/mweigand/.virtualenvs/numpy/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-99b71e71.so',
'internal_api': 'openblas',
'num_threads': 12,
'prefix': 'libscipy_openblas',
'threading_layer': 'pthreads',
'user_api': 'blas',
'version': '0.3.27'}]

Context for the issue:

For legacy code bases, sometimes only a small subset of functionality needs to be exported to python. Potentially, the described behavior prevents the successful generation of Python interfaces for those packages.

[HaoZeke]

Thanks for the detailed bug report and the suggested fix, would you be able to open a PR? If not I can incorporate this (with credit) and a test later this week.

[m-weigand]

Yes, I can prepare a PR, just wanted to make sure the fix is acceptable.

[HaoZeke]

Yes, I can prepare a PR, just wanted to make sure the fix is acceptable.

Awesome, yes I think this is a pragmatic solution and in keeping with the rest of the F2PY design, please feel free to ping me as a reviewer for the PR :)