| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-01 | |||
| 22:00:07 | openstack | Launchpad bug 1708031 in OpenStack Compute (nova) ""Failed to delete inventory for resource provider" error messages for predictable 409 cases" [Medium,Triaged] - Assigned to Matt Riedemann (mriedem) | |
| 22:00:07 | cdent | mriedem: does https://bugs.launchpad.net/nova/+bug/1708031 need an owner? If so, I can take it. Do we need the report client to be happy with both the old and new messages? Or just new? I’m guessing both. | |
| 22:01:11 | cdent | mriedem: nm, you assigned it to yourself while I was reading it | |
| 22:01:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix 409 handling in report client when deleting inventory https://review.openstack.org/489763 | |
| 22:01:41 | mriedem | cdent: ^ | |
| 22:07:53 | cdent | mriedem: has ironic been spewing these errors since early june? | |
| 22:08:09 | openstackgerrit | Michael Still proposed openstack/nova master: Read from console ptys using privsep. https://review.openstack.org/489486 | |
| 22:08:09 | openstackgerrit | Michael Still proposed openstack/nova master: Move execs of tee to privsep. https://review.openstack.org/489438 | |
| 22:08:20 | zzzeek | jaypipes: there? | |
| 22:08:40 | mriedem | cdent: probably | |
| 22:08:51 | mriedem | cdent: i only check the ironic ci job logs for ironic related chagnes | |
| 22:08:52 | mriedem | *changes | |
| 22:08:52 | cdent | mriedem: my real question: how come this didn’t break stuff ? | |
| 22:09:26 | zzzeek | mriedem: does nova get DB connection URLs from anywhere other than /etc/nova/nova.conf ? | |
| 22:10:23 | mriedem | zzzeek: yes | |
| 22:10:27 | mriedem | from the db for the cell mappings | |
| 22:10:31 | zzzeek | mriedem: in the DB ! | |
| 22:10:33 | mriedem | nova_api.cell_mappings table | |
| 22:10:34 | zzzeek | mriedem: omg. | |
| 22:10:49 | zzzeek | mriedem: so. once you deploy nova, you can never change the ip number of the database ?? | |
| 22:10:59 | zzzeek | ip number gets written into the DB itself ? | |
| 22:11:19 | mriedem | it's whatever is put in when creating the cell mapping | |
| 22:11:34 | melwitt | you can update the cell mapping | |
| 22:11:38 | melwitt | via nova-manage | |
| 22:11:56 | melwitt | if you need to change it | |
| 22:11:59 | zzzeek | melwitt: you can! did anyone consider maybe looking in the nova.conf file and seeing, oh hey they moved their DB server | |
| 22:12:08 | zzzeek | wow. | |
| 22:12:10 | zzzeek | nova. heh. | |
| 22:12:16 | zzzeek | OK | |
| 22:12:44 | mriedem | your volume connection info to a ceph server is also stored in a json blob in the db, so if your ceph server ip changes you have similar problems | |
| 22:12:51 | mriedem | per instance | |
| 22:12:58 | mriedem | so you have to migrate or reboot the instance | |
| 22:13:21 | zzzeek | WOW. OK wait, this actually destroys waht I want to do. wow | |
| 22:13:55 | mriedem | http://lists.openstack.org/pipermail/openstack-operators/2017-June/013700.html | |
| 22:14:26 | zzzeek | mriedem: im just looking at mysql urls | |
| 22:15:22 | colby_ | Hey Everyone I asked in the support channel but no one seemed to have an answer. We upgraded to newton and the flavor and aggregate migration scripts fail to cpoy the data to the nova_api tables. It appears that sqlalchecmy is adding the timezone offset to the DATETIME field and its complaining about incorrect format. Here is an example of the error: | |
| 22:15:23 | colby_ | Incorrect datetime value: '2015-02-17 22:37:06+00:00' for column 'created_at' | |
| 22:15:35 | zzzeek | mriedem: so confirm, it is completely impossible to have a scenario where a nova server on a particular controller had its own MySQL URL distinct from that of other contrlers, because they are pulled from the database and do not distinguish between the source | |
| 22:16:02 | colby_ | we use Mariadb 10.0.21 | |
| 22:17:03 | dansmith | zzzeek: I don't know what you mean by "between the source" but yes, all the control services find the cell databases by looking in the api database | |
| 22:17:10 | zzzeek | dansmith: OK | |
| 22:18:08 | melwitt | each controller has to know how to connect to each of N cell databases, that's why they're in the database, there are multiple DB URLs we need to be able to connect to | |
| 22:22:25 | mriedem | melwitt: +2 on your detach shenanigans | |
| 22:23:20 | melwitt | \o/ | |
| 22:25:33 | melwitt | mriedem: do you want me to move the comment inside the conditional? I can do that quick if you want to re +2 | |
| 22:25:50 | mriedem | meh | |
| 22:25:56 | melwitt | haha, okay | |
| 22:26:09 | mriedem | my ability to make any sort of decision right now is very low | |
| 22:27:11 | efried | morgan mordred sdague https://review.openstack.org/#/c/488137/ is passing tests with temporary patches to twiddle glance & ironic confs. | |
| 22:27:19 | mordred | efried: woot! | |
| 22:27:45 | efried | morgan mordred sdague And FYI, PS5 has the swizzle where you pass the service_type (rather than the conf_group/project_name) to the util. | |
| 22:27:55 | efried | (which both sdague and morgan asked for) | |
| 22:27:57 | morgan | nice | |
| 22:28:17 | efried | Next stop: cinder! | |
| 22:29:27 | mriedem | huh, we're actually failing grenade at a decent clip http://logs.openstack.org/54/489754/3/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/7e19ce3/logs/old/screen-n-cond.txt.gz#_2017-08-01_21_57_19_559 | |
| 22:29:38 | mriedem | on the ocata side | |
| 22:33:24 | openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | |
| 22:35:20 | cdent | mriedem: in muiltinode does one node stay ocata? is this before or after that? | |
| 22:36:02 | mordred | efried: a few suggestions inline | |
| 22:37:26 | mriedem | cdent: depends on the job config | |
| 22:37:39 | mriedem | some keep n-cpu behind | |
| 22:38:19 | mriedem | but yeah i think multinode grenade always keeps n-cpu behind as n-1 | |
| 22:38:22 | mriedem | and upgrades everything else | |
| 22:38:38 | mriedem | there used to be a single node grenade partial which upgraded everything except n-cpu | |
| 22:38:52 | mriedem | multinode grenade has 1 upgraded n-cpu and one n-1 n-cpu | |
| 22:39:06 | mriedem | so we can do things like live migrate between versions | |
| 22:39:18 | zzzeek | mriedem: crisis averted thanks for the help | |
| 22:41:29 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP Always use application/json accept header in report client https://review.openstack.org/489772 | |
| 22:41:51 | mriedem | zzzeek: can you shed some light on what the crisis was? | |
| 22:43:48 | zzzeek | mriedem: customer wants each controller accessing from its own source IP which is tied to a distinct password on the mysql server. because we use haproxy, that means each contorller must connect to a local haproxy, which means each controller has distinct database url | |
| 22:51:00 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_certs to Instance object https://review.openstack.org/489408 | |
| 22:51:42 | openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | |
| 22:58:46 | dansmith | mriedem: people like that can just put the url in the database using an alias hostname, that they map per node in /etc/hosts | |
| 22:58:48 | dansmith | mriedem: easy stuff | |
| 22:59:10 | dansmith | mriedem: and it appears that they can also configure the creds in a mysql client config file that gets honored, so they can have separate creds too | |
| 22:59:46 | dansmith | which is way better than us trying to pattern-format the url we get out of the db or something silly like that | |
| 23:01:12 | mriedem | but can postgresql do that?! | |
| 23:29:38 | gmann | mriedem, re: API doc patches. not yet. i will push those today | |
| 23:40:40 | gmann | mriedem, can you restore this - https://review.openstack.org/#/c/489641/1 | |
| 23:40:54 | gmann | api-microversion-history redirect is not done in https://review.openstack.org/#/c/489650/ | |
| 23:42:12 | mriedem | gmann: that's weird | |
| 23:42:13 | mriedem | done | |
| 23:44:30 | gmann | mriedem, thanks, it might need rebase once 489650 is in. i will do after main patch is merged | |
| 23:49:52 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add redirect for api-microversion-history doc https://review.openstack.org/489641 | |
| 23:54:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add redirect for api-microversion-history doc https://review.openstack.org/489641 | |
| #openstack-nova - 2017-08-02 | |||
| 00:16:33 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(1/2) https://review.openstack.org/408955 | |
| 00:24:43 | stewie925 | hello nova gurus, quick question? is there any rules about special characters in, say, flavor name or tenant name, etc? | |
| 00:29:44 | openstackgerrit | Michael Still proposed openstack/nova master: Avoid chowning console logs in libvirt https://review.openstack.org/472229 | |
| 00:29:45 | openstackgerrit | Michael Still proposed openstack/nova master: Move execs of touch to privsep. https://review.openstack.org/489190 | |
| 00:29:45 | openstackgerrit | Michael Still proposed openstack/nova master: First attempt at adding a privsep user to nova itself. https://review.openstack.org/459166 | |
| 00:29:46 | openstackgerrit | Michael Still proposed openstack/nova master: Move libvirt usages of chown to privsep. https://review.openstack.org/471972 | |
| 00:29:46 | openstackgerrit | Michael Still proposed openstack/nova master: Move execs of tee to privsep. https://review.openstack.org/489438 | |
| 00:29:47 | openstackgerrit | Michael Still proposed openstack/nova master: Refactor libvirt.utils.execute() away. https://review.openstack.org/489816 | |
| 00:29:47 | openstackgerrit | Michael Still proposed openstack/nova master: Read from console ptys using privsep. https://review.openstack.org/489486 | |
| 00:57:34 | openstackgerrit | Merged openstack/nova master: Add some more cellsv2 doc goodness https://review.openstack.org/487183 | |
| 00:58:41 | openstackgerrit | Merged openstack/nova master: always show urls in list_cells https://review.openstack.org/487860 | |
| 01:06:53 | openstackgerrit | Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766 | |
| 01:44:49 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_certs to Instance object https://review.openstack.org/489408 | |
| 01:48:29 | openstackgerrit | Merged openstack/nova master: add redirects for existing broken docs urls https://review.openstack.org/489650 | |
| 01:49:25 | openstackgerrit | Merged openstack/nova master: Deprecate cells v1 https://review.openstack.org/489632 | |