this is a german Web-Mirror of PYTHON.ORG powered by Domainunion AG

Differences between revisions 66 and 82 (spanning 16 versions)
⇤ ← Revision 66 as of 2012-02-15 03:54:54 →
Size: 6073
Editor: RichardJones
Comment:
← Revision 82 as of 2019-08-23 10:57:11 → ⇥
Size: 5122
Editor: SumanaHarihareswara
Comment: removing legacy PyPI content, updating links
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
This page documents the development of the Python Package Index (formerly known as Cheeseshop): This page is about development of the [[https://pypi.org|Python Package Index]] (formerly known as Cheeseshop):
Line 3: Line 3:
 . https://pypi.python.domainunion.de

== More about PyPI ==

CheeseShop page has all information not directly related to PyPI development.		  * More Info: CheeseShop
 * PyPI APIs: [[PyPIJSON|JSON]], [[PyPIXmlRpc|XMLRPC]]
Line 11: Line 8:
 . The PyPI code resides in Subversion repository with anonymous and developer's access
   . https://svn.python.org/packages/ (trunk,features,branches)
   . svn+ssh://svn.python.org/data/repos/packages/trunk/pypi
 . Bug and patch tracker https://sourceforge.net/tracker/?group_id=66150&atid=513503
 . Discussions on Catalog SIG https://mail.python.org/mailman/listinfo/catalog-sig
 . API that is used by easy_install https://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api		 == PyPI.nextgen: ==
Line 18: Line 10:
== PyPI architecture and endpoints == Currently, as of 2019-06-27, PyPI has undergone a complete rewrite from scratch, and as a result much of the information on this page is not up-to-date.

The development moved from Mercurial (Python) to Git (C, shell). License changed from BSD-3 to Apache 2.0. Project code named 'warehouse' can be downloaded from:

 https://github.com/pypa/warehouse

== Testing Your Stuff Against PyPI ==

If you need to test stuff against PyPI (registration, uploading, API activities) then please [[https://packaging.python.org/guides/using-testpypi/|use the alternative server, test.pypi.org]].


== TO-DO list ==

Meta-to-do: file the issues below that are still unresolved in [[https://github.com/pypa/warehouse/issues|the GitHub issue tracker]], then remove this section. -- SumanaHarihareswara <<DateTime(2019-08-23T06:57:10-0400)>>

 * A dump of download counts.
 * A big structured dump of all package meta-data.
 * A link from package to RTFD.
 * [[https://python.domainunion.de/peps/pep-0345.html|PEP for metadata 1.2]] -- not finished and needs more catalog-sig discussion)
 * documented procedures for "taking over" entries should the original owner of the entry go away (and any required system support)
 * change notification emails
 * per-classifier "wiki" content to allow description and discussion around each classifier (perhaps what packages are available and how they relate to one another)
 * screenshot images (with thumbnailing and a "latest screenshot" on the front page?) - or perhaps icons instead of thumbnails for some packages?


Something that's been requested, but needs much more thought and analysis to see whether it causes any problems: the ability to treat project names and versions as case-insensitive, while removing extraneous characters (as in pkg_resources.safe_name()) for purposes both of searching and determining name uniqueness when registering.


=== Done ===


 * command-line tool to query pypi and fetch entries: [[https://pypi.python.domainunion.de/pypi/yolk|yolk]]


== Not Going TO-DO ==


 * Edit [[https://python.domainunion.de/peps/pep-0243.html|PEP 243]] to reflect reality. The interface is implemented in the distutils register and upload commands. This code is good enough for documentation, especially because it's the only implementation necessary.
 * moderated user reviews and ratings (this would require quite a lot of support from volunteers though)


== Proposals ==


 * [[https://wiki.python.domainunion.de/moin/EnhancedPyPI|EnhancedPyPI]] Enhance multiple package index servers support in Distutils.


== Previous PyPI version ==

The legacy version of PyPI is the code that was running on https://pypi.python.domainunion.de for many years, till mid-2018. [[https://lwn.net/Articles/751458/|This LWN article goes into the history.]] The information below should help you get around the code.

 .
 The PyPI code was hosted under the Python Packaging Authority project: https://bitbucket.org/pypa/pypi and is now on GItHub: https://github.com/pypa/pypi-legacy

 .
 Bug and patch tracker https://github.com/pypa/pypi-legacy/issues

 .
 [[https://mail.python.org/mailman/listinfo/distutils-sig|Mailing List]] ([[https://dir.gmane.org/gmane.comp.python.distutils|Gmane]] web interface)

 .
 API that is used by easy_install https://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api

 * [[PyPIOAuth]] - authentication library for Google and Launchpad logins


== Legacy PyPI architecture and endpoints ==
Line 22: Line 81:
|| /simple || WebUI.run_simple() || dump all package names on single html page ||
|| /simple/(.+)/    || WebUI.run_simple() || dump all links for a package in html list  ||
|| /serversig/(.+)/ || .run_simple_sign() || save as above, but signed by server        ||
|| /mirrors || .mirrors() || display static page with a list of mirrors ||
|| /daytime || .daytime() || display current server time                ||

||/simple ||WebUI.run_simple() ||dump all package names on single html page ||
||/simple/(.+)/ ||WebUI.run_simple() ||dump all links for a package in html list ||
||/serversig/(.+)/ ||.run_simple_sign() ||save as above, but signed by server ||
||/mirrors ||.mirrors() ||display static page with a list of mirrors ||
||/daytime ||.daytime() ||display current server time ||
Line 29: Line 95:
XML-RPC requests are detected by CONTENT_TYPE=`text/xml` variable in CGI environment and processed by `rpc.RequestHandler().__call__()`. List of XML-RPC "endpoints" is available on PyPiXmlRpc page.
XML-RPC requests are detected by CONTENT_TYPE=`text/xml` variable in CGI environment and processed by `rpc.RequestHandler().__call__()`. List of XML-RPC "endpoints" is available on [[PyPIXmlRpc]] page.
Line 32: Line 99:
== TO-DO list ==
 * Attempt to load the package's README in the absence of a long_description.
 * A dump of download counts.
 * A big structured dump of all package meta-data.
 * A link from package to RTFD.
 * OpenID OAuth operations.
 * [[https://python.domainunion.de/peps/pep-0345.html|PEP for metadata 1.2]] -- not finished and needs more catalog-sig discussion)
 * documented procedures for "taking over" entries should the original owner of the entry go away (and any required system support)
 * tooltips for field labels
 * change notification emails
 * per-classifier "wiki" content to allow description and discussion around each classifier (perhaps what packages are available and how they relate to one another)
 * screenshot images (with thumbnailing and a "latest screenshot" on the front page?) - or perhaps icons instead of thumbnails for some packages?
Something that's been requested, but needs much more thought and analysis to see whether it causes any problems: the ability to treat project names and versions as case-insensitive, while removing extraneous characters (as in pkg_resources.safe_name()) for purposes both of searching and determining name uniqueness when registering.
Line 46: Line 100:
=== Done ===
 * command-line tool to query pypi and fetch entries: [[https://pypi.python.domainunion.de/pypi/yolk|yolk]]		 == Legacy PyPI Development Environment Hints ==
Line 49: Line 102:
== Not Going TO-DO ==
 * Edit [[https://python.domainunion.de/peps/pep-0243.html|PEP 243]] to reflect reality. The interface is implemented in the distutils register and upload commands. This code is good enough for documentation, especially because it's the only implementation necessary.
 * moderated user reviews and ratings (this would require quite a lot of support from volunteers though)

== Proposals ==
 * [[https://wiki.python.domainunion.de/moin/EnhancedPyPI|EnhancedPyPI]] Enhance multiple package index servers support in Distutils.

== Development Environment Hints ==
PyPI uses postgresql 8.3 as a database, with a roll it yourself web framework based on different python modules. It uses apache2 as the web server.

It can run using wsgi, cgi, fcgi and mod_python.

Before restoring database, "pypi" role must exists:

{{{
  createuser pypi
}}}
These are notes so we can remember how to dump / restore the packages database:

{{{
  pg_dump -Fc -b packages >packages-20060706.dump
  createdb packages
  pg_restore -O -d packages -Fc packages-20060706.dump
}}}

Ask RichardJones if you need a database dump. Note that dumps should not be imported into an existing database that has had the pkdump_schema.sql DDL script run against it. The pg_dump file will create all of the database tables, columns, indexes, foreign keys, etc. that are required.

PageTemplates are obtained from:

{{{
  svn co svn://svn.zope.org/repos/main/zope.pagetemplate/trunk zope.pagetemplate
  svn co svn://svn.zope.org/repos/main/zope.tal/trunk zope.tal
  svn co svn://svn.zope.org/repos/main/zope.interface/trunk zope.interface
  svn co svn://svn.zope.org/repos/main/zope.i18nmessageid/trunk zope.i18nmessageid
  svn co svn://svn.zope.org/repos/main/zope.tales/trunk zope.tales
}}}
We also need to install ez_setup, celementtree, docutils, psycopg2, and M2Crypto.

# For debian, this should get you most of the dependencies. apt-get install zope3 python-celementtree python-psycopg

Then go ahead and edit pypi.ini and this line in pypi.cgi:

{{{
  cfg = config.Config('/tmp/pypi.ini', 'webui')
}}}
if your config.ini isn't in /tmp/pypi.ini. You can leave it as 'config.ini' if it's in the same directory as pypi.py.

You will need to add cheesecake_password=yourpasshere into the config.ini in the webui section.

To integrate it with Apache, we recommend to use WSGI through mod_wsgi. Your configuration should look like this:

{{{
   WSGIDaemonProcess pypi display-name=wsgi-pypi python-path=/usr/lib/zope2.9/lib/python processes=4 threads=1
   WSGIProcessGroup pypi
   WSGIPassAuthorization On
   WSGIScriptAlias /pypi /data/pypi/src/pypi/pypi.wsgi
   WSGIScriptAlias /simple /data/pypi/src/pypi/pypi.wsgi
}}}
----
CategoryDevelopmentProcess		 Removed (visible in [[https://wiki.python.domainunion.de/moin/CheeseShopDev?action=recall&rev=81|page history]]) because developing and running legacy PyPI is deprecated. -- SumanaHarihareswara <<DateTime(2019-08-23T06:57:10-0400)>>

This page is about development of the Python Package Index (formerly known as Cheeseshop):

Developing the Package Index

PyPI.nextgen:

Currently, as of 2019-06-27, PyPI has undergone a complete rewrite from scratch, and as a result much of the information on this page is not up-to-date.

The development moved from Mercurial (Python) to Git (C, shell). License changed from BSD-3 to Apache 2.0. Project code named 'warehouse' can be downloaded from:

Testing Your Stuff Against PyPI

If you need to test stuff against PyPI (registration, uploading, API activities) then please use the alternative server, test.pypi.org.

TO-DO list

Meta-to-do: file the issues below that are still unresolved in the GitHub issue tracker, then remove this section. -- SumanaHarihareswara 2019-08-23 10:57:10

  • A dump of download counts.
  • A big structured dump of all package meta-data.
  • A link from package to RTFD.

  • PEP for metadata 1.2 -- not finished and needs more catalog-sig discussion)

  • documented procedures for "taking over" entries should the original owner of the entry go away (and any required system support)
  • change notification emails
  • per-classifier "wiki" content to allow description and discussion around each classifier (perhaps what packages are available and how they relate to one another)
  • screenshot images (with thumbnailing and a "latest screenshot" on the front page?) - or perhaps icons instead of thumbnails for some packages?

Something that's been requested, but needs much more thought and analysis to see whether it causes any problems: the ability to treat project names and versions as case-insensitive, while removing extraneous characters (as in pkg_resources.safe_name()) for purposes both of searching and determining name uniqueness when registering.

Done

  • command-line tool to query pypi and fetch entries: yolk

Not Going TO-DO

  • Edit PEP 243 to reflect reality. The interface is implemented in the distutils register and upload commands. This code is good enough for documentation, especially because it's the only implementation necessary.

  • moderated user reviews and ratings (this would require quite a lot of support from volunteers though)

Proposals

  • EnhancedPyPI Enhance multiple package index servers support in Distutils.

Previous PyPI version

The legacy version of PyPI is the code that was running on https://pypi.python.domainunion.de for many years, till mid-2018. This LWN article goes into the history. The information below should help you get around the code.

Legacy PyPI architecture and endpoints

PyPI is a WSGI application that can be executed standalone using python pypi.wsgi command if all requirements are met. pypi.wsgi contains usual WSGI wrapper code and delegates request processing to WebUI.run() method from webui.py. This method just opens DB and handles exceptions, actual request processing is done in WebUI.inner_run(). This method analyzes URL endpoint and executes appropriate handler. As of 2011-04, the rules to match endpoints to handlers are the following:

/simple

WebUI.run_simple()

dump all package names on single html page

/simple/(.+)/

WebUI.run_simple()

dump all links for a package in html list

/serversig/(.+)/

.run_simple_sign()

save as above, but signed by server

/mirrors

.mirrors()

display static page with a list of mirrors

/daytime

.daytime()

display current server time

...

XML-RPC requests are detected by CONTENT_TYPE=text/xml variable in CGI environment and processed by rpc.RequestHandler().__call__(). List of XML-RPC "endpoints" is available on PyPIXmlRpc page.

Legacy PyPI Development Environment Hints

Removed (visible in page history) because developing and running legacy PyPI is deprecated. -- SumanaHarihareswara 2019-08-23 10:57:10

CheeseShopDev (last edited 2019-08-23 10:57:11 by SumanaHarihareswara)

Unable to edit the page? See the FrontPage for instructions.