[GH-ISSUE #85] Implement PEP 621 (pyproject.toml) and migrate to "src" layout

kerem commented

2026-03-02 16:47:49 +03:00

Owner

Originally created by @buhtz on GitHub (Apr 24, 2023).
Original GitHub issue: https://github.com/initialcommit-com/git-sim/issues/85

Please consider to migrate your pyhton project to the last state of the art packaging process using pyproject.toml file only.

I'm aware that this might be a low prio issue currently.

Further reading:

Feel free to ask back if you have any questions.

Originally created by @buhtz on GitHub (Apr 24, 2023). Original GitHub issue: https://github.com/initialcommit-com/git-sim/issues/85 Please consider to migrate your pyhton project to the last state of the art packaging process using `pyproject.toml` file only. I'm aware that this might be a low prio issue currently. Further reading: - https://peps.python.org/pep-0621/ - https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html Feel free to ask back if you have any questions.

kerem closed this issue

2026-03-02 16:47:49 +03:00

kerem commented

2026-03-02 16:47:50 +03:00

Author

Owner

@buhtz commented on GitHub (Apr 24, 2023):

For two real world example see

This projects don't have setup.cfg or setup.py file and they do have the "src" layout.

@buhtz commented on GitHub (Apr 24, 2023): For two real world example see - https://codeberg.org/buhtz/hyperorg - https://codeberg.org/buhtz/buhtzology This projects don't have `setup.cfg` or `setup.py` file and they do have the "src" layout.

kerem commented

2026-03-02 16:47:50 +03:00

Author

Owner

@initialcommit-io commented on GitHub (May 9, 2023):

@buhtz Thx for the suggestion. This is something @paketb0te suggested in #9 (in addition to poetry), which I think you already saw.

I put this on the backburner since I don't have a lot of experience packaging Python projects and wanted to prioritize functionality before overhauling the structure with something I wasn't super familiar with.

Any chance you want to collaborate with @paketb0te on a PR for this?

Otherwise it will probably be a bit before I can brush up on it and do it myself.

@initialcommit-io commented on GitHub (May 9, 2023): @buhtz Thx for the suggestion. This is something @paketb0te suggested in #9 (in addition to poetry), which I think you already saw. I put this on the backburner since I don't have a lot of experience packaging Python projects and wanted to prioritize functionality before overhauling the structure with something I wasn't super familiar with. Any chance you want to collaborate with @paketb0te on a PR for this? Otherwise it will probably be a bit before I can brush up on it and do it myself.

kerem commented

2026-03-02 16:47:50 +03:00

Author

Owner

@paketb0te commented on GitHub (May 9, 2023):

@buhtz I am not sure I fully understand the advantages of using a src layout, even after reading THIS, THIS and the first half of the lengthy discussion HERE.

To me it seems like the main advantage is that "The src layout helps prevent accidental usage of the in-development copy of the code." - but I am not sure where / how that applies to our use case 🤔

Could you elaborate a bit why you propose the src layout and which advantages it brings?

Please note that I am not against this idea in any way, I just want to understand the reasons for it 😄

@paketb0te commented on GitHub (May 9, 2023): @buhtz I am not sure I fully understand the advantages of using a `src` layout, even after reading [THIS](https://hynek.me/articles/testing-packaging/), [THIS](https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/) and the first half of the lengthy discussion [HERE](https://github.com/pypa/packaging.python.org/issues/320). To me it seems like the main advantage is that "The src layout helps prevent accidental usage of the in-development copy of the code." - but I am not sure where / how that applies to our use case :thinking: Could you elaborate a bit why you propose the `src` layout and which advantages it brings? Please note that I am not against this idea in any way, I just want to understand the reasons for it :smile:

kerem commented

2026-03-02 16:47:50 +03:00

Author

Owner

@buhtz commented on GitHub (May 9, 2023):

Thanks for asking back. I think I told but I repeat (in my lame English) to be sure you understood.
I'm not a Python Pro and far away from having a deeper understanding of Python internals and design decisions.

btw: You can contact in me in German language if it might help.

My primary reason to use src-layout really is that the "Python Packing Authority" say so. It is the recommended way today.

understand the advantages of using a src layout

Me either. But what I understand from reading some things and in my own experience:

Most tools expect src layout or they can better handle a Python project repository if it is setup this way.
It clearer separate the code you want to ship as a release from the rest of your repository with helper scripts and all the other stuff you have.
With that layout it is also possible to have a "Python Distribution" (the official word for what we often call a package) with multiple "Import Packages". See bit_demo as an example. A "distribution" is what you would upload to PyPi or pack into a deb-file. A import package is what you have with import thepackage. The bit_demo tries to illustrate this. I haven't figured that out myself but had a lot of help from the community.
I never found myself or read about a disadvantage.

I'm sure that there are much more advantages. But they might be relevant only when you have bigger projects, with a CI pipeline and all that fancy stuff. I always tried to keep it as simple as possible. But my own projects are very tiny and there is no problem with that.

prevent accidental usage of the in-development copy of the code

Might be because of my English but I don't understand the term "in-development copy of the code".

Some short notes about pyproject.toml. I always had "problems" with the package meta data (version string, name, website url, stuff like this). I don't want to declare a version string on two different places. I want everything centralized in the toml file. But not all tools are able to handle that. But I also have IMHO elegant solution for this. In short my package does have a global meta dictionary which is generated from the available package metadata. A real example is my hyperorg project.

meta = _package_metadata_as_dict()
__version__ = meta['Version']
__name__ = meta['Name']

With that solution you can use everything you declared in your toml file in your python code. Not sure if there will come up a better solution from the python folks in the future. I have never shown that to a python pro, e.g. on their mailinglist.

@buhtz commented on GitHub (May 9, 2023): Thanks for asking back. I think I told but I repeat (in my lame English) to be sure you understood. I'm not a Python Pro and far away from having a deeper understanding of Python internals and design decisions. btw: You can contact in me in German language if it might help. My primary reason to use src-layout really is that the "Python Packing Authority" say so. It is the recommended way today. > understand the advantages of using a `src` layout Me either. But what I understand from reading some things and in my own experience: - Most tools expect src layout or they can better handle a Python project repository if it is setup this way. - It clearer separate the code you want to ship as a release from the rest of your repository with helper scripts and all the other stuff you have. - With that layout it is also possible to have a "Python Distribution" (the official word for what we often call a package) with multiple "Import Packages". See [`bit_demo`](https://codeberg.org/buhtz/bit_demo) as an example. A "distribution" is what you would upload to PyPi or pack into a deb-file. A import package is what you have with `import thepackage`. The `bit_demo` tries to illustrate this. I haven't figured that out myself but had a lot of help from the community. - I never found myself or read about a disadvantage. I'm sure that there are much more advantages. But they might be relevant only when you have bigger projects, with a CI pipeline and all that fancy stuff. I always tried to keep it as simple as possible. But my own projects are very tiny and there is no problem with that. > prevent accidental usage of the in-development copy of the code Might be because of my English but I don't understand the term "in-development copy of the code". Some short notes about pyproject.toml. I always had "problems" with the package meta data (version string, name, website url, stuff like this). I don't want to declare a version string on two different places. I want everything centralized in the toml file. But not all tools are able to handle that. But I also have IMHO elegant solution for this. In short my package does have a global `meta` dictionary which is generated from the available package metadata. A real example is my [`hyperorg`](https://codeberg.org/buhtz/hyperorg/src/branch/develop/src/hyperorg/__init__.py) project. ``` meta = _package_metadata_as_dict() __version__ = meta['Version'] __name__ = meta['Name'] ``` With that solution you can use everything you declared in your toml file in your python code. Not sure if there will come up a better solution from the python folks in the future. I have never shown that to a python pro, e.g. on their mailinglist.

kerem commented

2026-03-02 16:47:51 +03:00

Author

Owner

@paketb0te commented on GitHub (May 10, 2023):

@buhtz thanks for the detailed response.
Maybe we can do it step by step, start by migrating from setup.py to pyproject.toml and later change to the src-layout if we find it to be a good tradeoff of simplicity vs. "less room for errors".

@paketb0te commented on GitHub (May 10, 2023): @buhtz thanks for the detailed response. Maybe we can do it step by step, start by migrating from `setup.py` to `pyproject.toml` and later change to the src-layout if we find it to be a good tradeoff of simplicity vs. "less room for errors".

kerem commented

2026-03-02 16:47:51 +03:00

Author

Owner

@buhtz commented on GitHub (May 10, 2023):

I see no need to separate that steps but I also see no argument against separating them.

I would suggest move to src-layout first. That also includes moving you unittest files into a separate folder called "test" or "tests".

@buhtz commented on GitHub (May 10, 2023): I see no need to separate that steps but I also see no argument against separating them. I would suggest move to src-layout first. That also includes moving you unittest files into a separate folder called "test" or "tests".

kerem commented

2026-03-02 16:47:51 +03:00

Author

Owner

@initialcommit-io commented on GitHub (Jul 10, 2023):

Hello again @paketb0te and @buhtz. When packaging the most recent release of git-sim and trying to upload to PyPI I started running into some errors (didn't save them unfortunately) that were due to the way git-sim is packaged. Like you both mentioned looks like pyproject.toml is the new supported way and using the setup.py is basically being deprecated.

I was able to get around this by downgrading setuptools to version 58.2.0, but I think this is a big sign we should migrate the project now so that we can use the most up to date tools going forward.

After reading the details of the "src" layout, it seems to be essentially the recommended way by PyPA, and since we already use editable install for our development process anyway, doesn't seem like there is a downside.

Would either of you like to work on this?

I would suggest move to src-layout first. That also includes moving you unittest files into a separate folder called "test" or "tests".

FYI the test suite and our unit test stub have already been moved into a tests/ folder.

@initialcommit-io commented on GitHub (Jul 10, 2023): Hello again @paketb0te and @buhtz. When packaging the most recent release of git-sim and trying to upload to PyPI I started running into some errors (didn't save them unfortunately) that were due to the way git-sim is packaged. Like you both mentioned looks like pyproject.toml is the new supported way and using the setup.py is basically being deprecated. I was able to get around this by downgrading setuptools to version 58.2.0, but I think this is a big sign we should migrate the project now so that we can use the most up to date tools going forward. After reading the details of the "src" layout, it seems to be essentially the recommended way by PyPA, and since we already use editable install for our development process anyway, doesn't seem like there is a downside. Would either of you like to work on this? > I would suggest move to src-layout first. That also includes moving you unittest files into a separate folder called "test" or "tests". FYI the test suite and our unit test stub have already been moved into a `tests/` folder.

kerem commented

2026-03-02 16:47:51 +03:00

Author

Owner

@buhtz commented on GitHub (Jul 10, 2023):

Dear initialcommit-io ,
thanks for trying out.

I won't have resource in the next months. But I can help analyzing problems.

Maybe you can try to reproduce your problem and offer your setup in a separate branch .Then I can have a look.

@buhtz commented on GitHub (Jul 10, 2023): Dear initialcommit-io , thanks for trying out. I won't have resource in the next months. But I can help analyzing problems. Maybe you can try to reproduce your problem and offer your setup in a separate branch .Then I can have a look.

kerem commented

2026-03-02 16:47:51 +03:00

Author

Owner

@initialcommit-io commented on GitHub (Jul 10, 2023):

Thank you. But I have a workaround for the issue for now, so I think we should spend more time actually doing the project.toml and src structure refactorings. @paketb0te Any chance you're interested in helping with this since you had originally mentioned it a few months ago? 😄

@initialcommit-io commented on GitHub (Jul 10, 2023): Thank you. But I have a workaround for the issue for now, so I think we should spend more time actually doing the project.toml and src structure refactorings. @paketb0te Any chance you're interested in helping with this since you had originally mentioned it a few months ago? 😄

kerem commented

2026-03-02 16:47:51 +03:00

Author

Owner

@paketb0te commented on GitHub (Jul 11, 2023):

@initialcommit-io yeah, I can take a stab at it 😃

If there is any bigger work in progress, it would probably be smart to integrate / merge that into the dev branch before restructuring, otherwise I think the (later) merging will be more difficult then neccessary.

@paketb0te commented on GitHub (Jul 11, 2023): @initialcommit-io yeah, I can take a stab at it :smiley: If there is any bigger work in progress, it would probably be smart to integrate / merge that into the dev branch before restructuring, otherwise I think the (later) merging will be more difficult then neccessary.

kerem commented

2026-03-02 16:47:51 +03:00

Author

Owner

@initialcommit-io commented on GitHub (Jul 12, 2023):

Thanks @paketb0te !!!

Yes the dev branch should be up to date with the latest changes. Mostly its stuff related to the e2e test suite, which could be useful to run for the first time in a real situation after making your changes 😸

@initialcommit-io commented on GitHub (Jul 12, 2023): Thanks @paketb0te !!! Yes the dev branch should be up to date with the latest changes. Mostly its stuff related to the e2e test suite, which could be useful to run for the first time in a real situation after making your changes 😸

kerem commented

2026-03-02 16:47:51 +03:00

Author

Owner

@paketb0te commented on GitHub (Jul 14, 2023):

@initialcommit-io there is from PIL import Image, ImageChops in tests/e2e_tests/utils.py, what package is that?
pillow?

@paketb0te commented on GitHub (Jul 14, 2023): @initialcommit-io there is `from PIL import Image, ImageChops` in `tests/e2e_tests/utils.py`, what package is that? pillow?

kerem commented

2026-03-02 16:47:51 +03:00

Author

Owner

@initialcommit-io commented on GitHub (Jul 14, 2023):

there is from PIL import Image, ImageChops in tests/e2e_tests/utils.py, what package is that?
pillow?

Yup, should be!

@initialcommit-io commented on GitHub (Jul 14, 2023): > there is from PIL import Image, ImageChops in tests/e2e_tests/utils.py, what package is that? pillow? Yup, should be!

kerem commented

2026-03-02 16:47:51 +03:00

Author

Owner

@initialcommit-io commented on GitHub (Jul 17, 2023):

Closing since the migration to src layout and pyproject.toml was successful! Thanks again @paketb0te for making this happen.

@initialcommit-io commented on GitHub (Jul 17, 2023): Closing since the migration to src layout and pyproject.toml was successful! Thanks again @paketb0te for making this happen.

kerem referenced this issue

2026-03-02 16:48:07 +03:00

[PR #54] [MERGED] add trailing underscore to env_prefix #85

Rows
Columns

[GH-ISSUE #85] Implement PEP 621 (pyproject.toml) and migrate to "src" layout #54