Size: 3208
Comment:
|
Size: 6989
Comment: shorten intro, add links to apis
|
Deletions are marked like this. | Additions are marked like this. |
Line 1: | Line 1: |
This page documents the development of the Python Cheese Shop: | This page is about development of [[https://pypi.python.org|Python Package Index]] (formerly known as Cheeseshop): |
Line 3: | Line 3: |
https://cheeseshop.python.org | * info: CheeseShop * apis: [[PyPIJSON|JSON]], [[PyPIXmlRpc|XMLRPC]] |
Line 5: | Line 6: |
The PyPI (packages index, Cheese Shop web interface) code resides in SVN at: | = Developing the Package Index = |
Line 7: | Line 8: |
https://svn.python.org/packages/ (trunk,features,branches) | == PyPI.nextgen: == |
Line 9: | Line 10: |
Requests for new Trove categories: | Currently, as of 2013-11-11, PyPI is undergoing a complete rewrite from scratch, and as a result much of the information on this page is not actual. You can see preview of the new site at: |
Line 11: | Line 12: |
Python frameworks with plugins or packages that target the framework can get their own category. The category should only be added *after* such packages exist. Complimentary packages can link to each other from their descriptions, they do not need a category to link them together; only when packages are provided by different people does a category need to be created. To ask for a category email catalog-sig@python.org. |
https://preview-pypi.python.org/ |
Line 19: | Line 14: |
Patches and bug reports should be submitted to: | The development moved from Mercurial (Python) to Git (C, shell). License changed from BSD-3 to Apache 2.0. Project code named 'warehouse' can be downloaded from: |
Line 21: | Line 16: |
https://sourceforge.net/tracker/?group_id=66150&atid=513503 | https://github.com/pypa/warehouse |
Line 23: | Line 18: |
Discussion about the Cheese Shop should be had on the Catalog SIG mailing list: | |
Line 25: | Line 19: |
https://mail.python.org/mailman/listinfo/catalog-sig | == Previous PyPI version == |
Line 27: | Line 21: |
== XML-RPC Interface == | "previous" version of PyPI is the code that was running on https://pypi.python.domainunion.de from ... till the end of 2013. It was originally written by ... and was running on ... . The information below should help you get around the code. |
Line 29: | Line 23: |
The Cheese Shop has an XML-RPC interface. See CheeseShopXmlRpc for details. | . The PyPI code is hosted under the Python Packaging Authority project: https://bitbucket.org/pypa/pypi . Bug and patch tracker https://bitbucket.org/pypa/pypi/issues . [[https://mail.python.org/mailman/listinfo/distutils-sig|Mailing List]] ([[https://dir.gmane.org/gmane.comp.python.distutils|Gmane]] web interface) . API that is used by easy_install https://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api * [[PyPIOAuth]] - authentication library for Google and Launchpad logins == PyPI architecture and endpoints == PyPI is a WSGI application that can be executed standalone using `python pypi.wsgi` command if all requirements are met. `pypi.wsgi` contains usual WSGI wrapper code and delegates request processing to `WebUI.run()` method from `webui.py`. This method just opens DB and handles exceptions, actual request processing is done in `WebUI.inner_run()`. This method analyzes URL endpoint and executes appropriate handler. As of 2011-04, the rules to match endpoints to handlers are the following: ||/simple ||WebUI.run_simple() ||dump all package names on single html page || ||/simple/(.+)/ ||WebUI.run_simple() ||dump all links for a package in html list || ||/serversig/(.+)/ ||.run_simple_sign() ||save as above, but signed by server || ||/mirrors ||.mirrors() ||display static page with a list of mirrors || ||/daytime ||.daytime() ||display current server time || ... XML-RPC requests are detected by CONTENT_TYPE=`text/xml` variable in CGI environment and processed by `rpc.RequestHandler().__call__()`. List of XML-RPC "endpoints" is available on [[PyPIXmlRpc]] page. == Testing Your Stuff Against PyPI == If you need to test stuff against PyPI (registration, uploading, API activities) then please use the alternative server "testpypi.python.org". |
Line 33: | Line 70: |
* [https://python.domainunion.de/peps/pep-0345.html PEP for metadata 1.2] -- not finished and needs more catalog-sig discussion) * edit [https://python.domainunion.de/peps/pep-0243.html PEP 243] to reflect reality * auto-generate download_urls for package uploads * command-line tool to query pypi and fetch entries * DOAP support |
* A dump of download counts. * A big structured dump of all package meta-data. * A link from package to RTFD. * [[https://python.domainunion.de/peps/pep-0345.html|PEP for metadata 1.2]] -- not finished and needs more catalog-sig discussion) |
Line 39: | Line 76: |
* better searching (eg. single search box on the front page) | |
Line 42: | Line 78: |
* auto-compute [https://pycheesecake.org Cheesecake] scores * moderated user reviews and ratings * strip leading whitespace from description before ReST |
* per-classifier "wiki" content to allow description and discussion around each classifier (perhaps what packages are available and how they relate to one another) * screenshot images (with thumbnailing and a "latest screenshot" on the front page?) - or perhaps icons instead of thumbnails for some packages? |
Line 46: | Line 81: |
Something that's been requested, but needs much more thought and analysis to see whether it causes any problems: the ability to treat project names and versions as case-insensitive, while removing extraneous characters (as in pkg_resources.safe_name()) for purposes both of searching and determining name uniqueness when registering. |
Something that's been requested, but needs much more thought and analysis to see whether it causes any problems: the ability to treat project names and versions as case-insensitive, while removing extraneous characters (as in pkg_resources.safe_name()) for purposes both of searching and determining name uniqueness when registering. === Done === * command-line tool to query pypi and fetch entries: [[https://pypi.python.domainunion.de/pypi/yolk|yolk]] == Not Going TO-DO == * Edit [[https://python.domainunion.de/peps/pep-0243.html|PEP 243]] to reflect reality. The interface is implemented in the distutils register and upload commands. This code is good enough for documentation, especially because it's the only implementation necessary. * moderated user reviews and ratings (this would require quite a lot of support from volunteers though) == Proposals == * [[https://wiki.python.domainunion.de/moin/EnhancedPyPI|EnhancedPyPI]] Enhance multiple package index servers support in Distutils. |
Line 52: | Line 106: |
PyPI uses postgresql 8.3 as a database, with a roll it yourself web framework based on different python modules. It uses apache2 as the web server. It can run using wsgi, cgi, fcgi and mod_python. Before restoring database, "pypi" role must exists: {{{ createuser pypi }}} |
|
Line 54: | Line 126: |
{{{ |
|
Line 55: | Line 132: |
createdb packages pg_restore -O -d packages -Fc packages-20060706.dump }}} |
|
Line 56: | Line 136: |
createdb packages | |
Line 58: | Line 137: |
pg_restore -O -d packages -Fc packages-20060706.dump | Ask [[RichardJones|RichardJones]] if you need a database dump. Note that dumps should not be imported into an existing database that has had the pkdump_schema.sql DDL script run against it. The pg_dump file will create all of the database tables, columns, indexes, foreign keys, etc. that are required. |
Line 60: | Line 139: |
Ask RichardJones if you need a database dump. | |
Line 62: | Line 140: |
PageTemplates are obtained from: | [[PageTemplates|PageTemplates]] are obtained from: |
Line 64: | Line 142: |
svn co svn://svn.zope.org/repos/main/zope.pagetemplate/trunk zope. pagetemplate | |
Line 66: | Line 143: |
{{{ svn co svn://svn.zope.org/repos/main/zope.pagetemplate/trunk zope.pagetemplate |
|
Line 67: | Line 149: |
svn co svn://svn.zope.org/repos/main/zope.interface/trunk zope.interface svn co svn://svn.zope.org/repos/main/zope.i18nmessageid/trunk zope.i18nmessageid svn co svn://svn.zope.org/repos/main/zope.tales/trunk zope.tales }}} |
|
Line 68: | Line 154: |
svn co svn://svn.zope.org/repos/main/zope.interface/trunk zope.interface | |
Line 70: | Line 155: |
svn co svn://svn.zope.org/repos/main/zope.i18nmessageid/trunk zope.i18nmessageid | We also need to install ez_setup, celementtree, docutils, psycopg2, and [[M2Crypto|M2Crypto]]. |
Line 72: | Line 157: |
svn co svn://svn.zope.org/repos/main/zope.tales/trunk zope.tales | |
Line 74: | Line 158: |
We also need to install ez_setup and docutils. | # For debian, this should get you most of the dependencies. apt-get install zope3 python-celementtree python-psycopg |
Line 76: | Line 160: |
Then go ahead and edit pypi.ini and this line in pypi.py: | |
Line 78: | Line 161: |
Then go ahead and edit pypi.ini and this line in pypi.cgi: {{{ |
|
Line 79: | Line 169: |
}}} | |
Line 80: | Line 171: |
if your config.ini isn't in /tmp/pypi.ini. You can leave it as 'config.ini' if it's in the same directory as pypi.py. | if your config.ini isn't in /tmp/pypi.ini. You can leave it as 'config.ini' if it's in the same directory as pypi.py. You will need to add cheesecake_password=yourpasshere into the config.ini in the webui section. To integrate it with Apache, we recommend to use WSGI through mod_wsgi. Your configuration should look like this: {{{ WSGIDaemonProcess pypi display-name=wsgi-pypi python-path=/usr/lib/zope2.9/lib/python processes=4 threads=1 WSGIProcessGroup pypi WSGIPassAuthorization On WSGIScriptAlias /pypi /data/pypi/src/pypi/pypi.wsgi WSGIScriptAlias /simple /data/pypi/src/pypi/pypi.wsgi }}} ---- [[CategoryDevelopmentProcess|CategoryDevelopmentProcess]] |
This page is about development of Python Package Index (formerly known as Cheeseshop):
info: CheeseShop
Developing the Package Index
PyPI.nextgen:
Currently, as of 2013-11-11, PyPI is undergoing a complete rewrite from scratch, and as a result much of the information on this page is not actual. You can see preview of the new site at:
The development moved from Mercurial (Python) to Git (C, shell). License changed from BSD-3 to Apache 2.0. Project code named 'warehouse' can be downloaded from:
Previous PyPI version
"previous" version of PyPI is the code that was running on https://pypi.python.domainunion.de from ... till the end of 2013. It was originally written by ... and was running on ... . The information below should help you get around the code.
The PyPI code is hosted under the Python Packaging Authority project: https://bitbucket.org/pypa/pypi
Bug and patch tracker https://bitbucket.org/pypa/pypi/issues
Mailing List (Gmane web interface)
API that is used by easy_install https://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api
PyPIOAuth - authentication library for Google and Launchpad logins
PyPI architecture and endpoints
PyPI is a WSGI application that can be executed standalone using python pypi.wsgi command if all requirements are met. pypi.wsgi contains usual WSGI wrapper code and delegates request processing to WebUI.run() method from webui.py. This method just opens DB and handles exceptions, actual request processing is done in WebUI.inner_run(). This method analyzes URL endpoint and executes appropriate handler. As of 2011-04, the rules to match endpoints to handlers are the following:
/simple |
WebUI.run_simple() |
dump all package names on single html page |
/simple/(.+)/ |
WebUI.run_simple() |
dump all links for a package in html list |
/serversig/(.+)/ |
.run_simple_sign() |
save as above, but signed by server |
/mirrors |
.mirrors() |
display static page with a list of mirrors |
/daytime |
.daytime() |
display current server time |
...
XML-RPC requests are detected by CONTENT_TYPE=text/xml variable in CGI environment and processed by rpc.RequestHandler().__call__(). List of XML-RPC "endpoints" is available on PyPIXmlRpc page.
Testing Your Stuff Against PyPI
If you need to test stuff against PyPI (registration, uploading, API activities) then please use the alternative server "testpypi.python.org".
TO-DO list
- A dump of download counts.
- A big structured dump of all package meta-data.
- A link from package to RTFD.
PEP for metadata 1.2 -- not finished and needs more catalog-sig discussion)
- documented procedures for "taking over" entries should the original owner of the entry go away (and any required system support)
- tooltips for field labels
- change notification emails
- per-classifier "wiki" content to allow description and discussion around each classifier (perhaps what packages are available and how they relate to one another)
- screenshot images (with thumbnailing and a "latest screenshot" on the front page?) - or perhaps icons instead of thumbnails for some packages?
Something that's been requested, but needs much more thought and analysis to see whether it causes any problems: the ability to treat project names and versions as case-insensitive, while removing extraneous characters (as in pkg_resources.safe_name()) for purposes both of searching and determining name uniqueness when registering.
Done
command-line tool to query pypi and fetch entries: yolk
Not Going TO-DO
Edit PEP 243 to reflect reality. The interface is implemented in the distutils register and upload commands. This code is good enough for documentation, especially because it's the only implementation necessary.
- moderated user reviews and ratings (this would require quite a lot of support from volunteers though)
Proposals
EnhancedPyPI Enhance multiple package index servers support in Distutils.
Development Environment Hints
PyPI uses postgresql 8.3 as a database, with a roll it yourself web framework based on different python modules. It uses apache2 as the web server.
It can run using wsgi, cgi, fcgi and mod_python.
Before restoring database, "pypi" role must exists:
createuser pypi
These are notes so we can remember how to dump / restore the packages database:
pg_dump -Fc -b packages >packages-20060706.dump createdb packages pg_restore -O -d packages -Fc packages-20060706.dump
Ask RichardJones if you need a database dump. Note that dumps should not be imported into an existing database that has had the pkdump_schema.sql DDL script run against it. The pg_dump file will create all of the database tables, columns, indexes, foreign keys, etc. that are required.
PageTemplates are obtained from:
svn co svn://svn.zope.org/repos/main/zope.pagetemplate/trunk zope.pagetemplate svn co svn://svn.zope.org/repos/main/zope.tal/trunk zope.tal svn co svn://svn.zope.org/repos/main/zope.interface/trunk zope.interface svn co svn://svn.zope.org/repos/main/zope.i18nmessageid/trunk zope.i18nmessageid svn co svn://svn.zope.org/repos/main/zope.tales/trunk zope.tales
We also need to install ez_setup, celementtree, docutils, psycopg2, and M2Crypto.
# For debian, this should get you most of the dependencies. apt-get install zope3 python-celementtree python-psycopg
Then go ahead and edit pypi.ini and this line in pypi.cgi:
cfg = config.Config('/tmp/pypi.ini', 'webui')
if your config.ini isn't in /tmp/pypi.ini. You can leave it as 'config.ini' if it's in the same directory as pypi.py.
You will need to add cheesecake_password=yourpasshere into the config.ini in the webui section.
To integrate it with Apache, we recommend to use WSGI through mod_wsgi. Your configuration should look like this:
WSGIDaemonProcess pypi display-name=wsgi-pypi python-path=/usr/lib/zope2.9/lib/python processes=4 threads=1 WSGIProcessGroup pypi WSGIPassAuthorization On WSGIScriptAlias /pypi /data/pypi/src/pypi/pypi.wsgi WSGIScriptAlias /simple /data/pypi/src/pypi/pypi.wsgi