| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-01 | |||
| 21:24:02 | dansmith | so that'll fail in our current multinode setup right? | |
| 21:24:23 | mriedem | that's what i'm trying to test | |
| 21:24:43 | dansmith | well, it'll be flaky I think | |
| 21:25:13 | mriedem | b/c we don't have the upcall? | |
| 21:25:29 | dansmith | yeah | |
| 21:25:30 | dansmith | well, | |
| 21:25:35 | dansmith | hmm | |
| 21:25:38 | mriedem | does that scenario rely on rebuilds? | |
| 21:25:42 | dansmith | no | |
| 21:26:01 | dansmith | it requires scheduler races | |
| 21:26:18 | mriedem | i think it should just say server 1 is going on host A and when the filter scheduler processes server 2, it should put it on host B because we're tracking server1 on A in the HostState | |
| 21:26:46 | mriedem | i'm really just trying to make sure the HostState stuff doesn't explode w/o the upcalls | |
| 21:27:36 | dansmith | right, but your test may validate that without enough parallelism the right thing happens early on | |
| 21:27:41 | dansmith | which I guess is what you're going for, | |
| 21:28:01 | dansmith | but it doesn't prove that the upcall is working (especially since it won't even be done currently) | |
| 21:28:14 | mriedem | yeah not trying to prove the upcall | |
| 21:28:29 | mriedem | just trying to make sure the basic scenario through the filter works properly | |
| 21:29:02 | mriedem | i think people have tried to add a test for this in tempest in the past and it's been shot down because it could be done in nova functional tests, but i worry about how much we actually stub out in the functional stuff | |
| 21:29:15 | dansmith | I guess I'm not sure how the filter really works so I can't say if it'll be likely to fail | |
| 21:29:25 | dansmith | yeah | |
| 21:29:29 | mikal | Morning | |
| 21:29:45 | cfriesen | was there an official notification email that the Sydney presentation voting was open? I don't remember seeing one. | |
| 21:30:13 | mriedem | i don't remember seeing one, but i don't vote either | |
| 21:30:23 | mriedem | i'm only one person, | |
| 21:30:25 | mriedem | my vote doesn't count | |
| 21:32:54 | cdent | cfriesen: I’ve seen a few different announcements, but I’m not sure on which lists | |
| 21:33:11 | cdent | and there were some “we’ve extended voiting” announcements too | |
| 21:33:17 | openstackgerrit | Ed Leafe proposed openstack/nova master: Handle addition of new nodes/instances in ironic flavor migration https://review.openstack.org/487954 | |
| 21:36:12 | melwitt | I can't find a link to the voting other than the one that cisco person sent out | |
| 21:36:25 | melwitt | (can't find in email I mean) | |
| 21:38:56 | mriedem | edleafe: i think i'm going to make us run E128 just for you https://review.openstack.org/#/c/487925/2/nova/virt/ironic/driver.py | |
| 21:43:43 | cdent | melwitt, cfriesen: I’m on a _lot_ of lists, so may not have been os-dev | |
| 21:44:07 | melwitt | I found one email to os-dev but it had no link to voting :P | |
| 21:44:25 | melwitt | vote ... somewhere ... that you should find by googling | |
| 21:45:23 | edleafe | mriedem: I'm consistent :) | |
| 21:54:29 | cfriesen | mriedem: for https://review.openstack.org/#/c/339715 would you be okay with me updating the "stale" migration to a "failed" or "error" state when we detect it? Or just check for them at nova-compute startup and assume they can't happen during "normal" running? | |
| 21:57:53 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_certs to Instance object https://review.openstack.org/489408 | |
| 21:58:03 | jaypipes | mriedem: marked that one patch -2. | |
| 22:00:07 | openstack | Launchpad bug 1708031 in OpenStack Compute (nova) ""Failed to delete inventory for resource provider" error messages for predictable 409 cases" [Medium,Triaged] - Assigned to Matt Riedemann (mriedem) | |
| 22:00:07 | cdent | mriedem: does https://bugs.launchpad.net/nova/+bug/1708031 need an owner? If so, I can take it. Do we need the report client to be happy with both the old and new messages? Or just new? I’m guessing both. | |
| 22:01:11 | cdent | mriedem: nm, you assigned it to yourself while I was reading it | |
| 22:01:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix 409 handling in report client when deleting inventory https://review.openstack.org/489763 | |
| 22:01:41 | mriedem | cdent: ^ | |
| 22:07:53 | cdent | mriedem: has ironic been spewing these errors since early june? | |
| 22:08:09 | openstackgerrit | Michael Still proposed openstack/nova master: Read from console ptys using privsep. https://review.openstack.org/489486 | |
| 22:08:09 | openstackgerrit | Michael Still proposed openstack/nova master: Move execs of tee to privsep. https://review.openstack.org/489438 | |
| 22:08:20 | zzzeek | jaypipes: there? | |
| 22:08:40 | mriedem | cdent: probably | |
| 22:08:51 | mriedem | cdent: i only check the ironic ci job logs for ironic related chagnes | |
| 22:08:52 | mriedem | *changes | |
| 22:08:52 | cdent | mriedem: my real question: how come this didn’t break stuff ? | |
| 22:09:26 | zzzeek | mriedem: does nova get DB connection URLs from anywhere other than /etc/nova/nova.conf ? | |
| 22:10:23 | mriedem | zzzeek: yes | |
| 22:10:27 | mriedem | from the db for the cell mappings | |
| 22:10:31 | zzzeek | mriedem: in the DB ! | |
| 22:10:33 | mriedem | nova_api.cell_mappings table | |
| 22:10:34 | zzzeek | mriedem: omg. | |
| 22:10:49 | zzzeek | mriedem: so. once you deploy nova, you can never change the ip number of the database ?? | |
| 22:10:59 | zzzeek | ip number gets written into the DB itself ? | |
| 22:11:19 | mriedem | it's whatever is put in when creating the cell mapping | |
| 22:11:34 | melwitt | you can update the cell mapping | |
| 22:11:38 | melwitt | via nova-manage | |
| 22:11:56 | melwitt | if you need to change it | |
| 22:11:59 | zzzeek | melwitt: you can! did anyone consider maybe looking in the nova.conf file and seeing, oh hey they moved their DB server | |
| 22:12:08 | zzzeek | wow. | |
| 22:12:10 | zzzeek | nova. heh. | |
| 22:12:16 | zzzeek | OK | |
| 22:12:44 | mriedem | your volume connection info to a ceph server is also stored in a json blob in the db, so if your ceph server ip changes you have similar problems | |
| 22:12:51 | mriedem | per instance | |
| 22:12:58 | mriedem | so you have to migrate or reboot the instance | |
| 22:13:21 | zzzeek | WOW. OK wait, this actually destroys waht I want to do. wow | |
| 22:13:55 | mriedem | http://lists.openstack.org/pipermail/openstack-operators/2017-June/013700.html | |
| 22:14:26 | zzzeek | mriedem: im just looking at mysql urls | |
| 22:15:22 | colby_ | Hey Everyone I asked in the support channel but no one seemed to have an answer. We upgraded to newton and the flavor and aggregate migration scripts fail to cpoy the data to the nova_api tables. It appears that sqlalchecmy is adding the timezone offset to the DATETIME field and its complaining about incorrect format. Here is an example of the error: | |
| 22:15:23 | colby_ | Incorrect datetime value: '2015-02-17 22:37:06+00:00' for column 'created_at' | |
| 22:15:35 | zzzeek | mriedem: so confirm, it is completely impossible to have a scenario where a nova server on a particular controller had its own MySQL URL distinct from that of other contrlers, because they are pulled from the database and do not distinguish between the source | |
| 22:16:02 | colby_ | we use Mariadb 10.0.21 | |
| 22:17:03 | dansmith | zzzeek: I don't know what you mean by "between the source" but yes, all the control services find the cell databases by looking in the api database | |
| 22:17:10 | zzzeek | dansmith: OK | |
| 22:18:08 | melwitt | each controller has to know how to connect to each of N cell databases, that's why they're in the database, there are multiple DB URLs we need to be able to connect to | |
| 22:22:25 | mriedem | melwitt: +2 on your detach shenanigans | |
| 22:23:20 | melwitt | \o/ | |
| 22:25:33 | melwitt | mriedem: do you want me to move the comment inside the conditional? I can do that quick if you want to re +2 | |
| 22:25:50 | mriedem | meh | |
| 22:25:56 | melwitt | haha, okay | |
| 22:26:09 | mriedem | my ability to make any sort of decision right now is very low | |
| 22:27:11 | efried | morgan mordred sdague https://review.openstack.org/#/c/488137/ is passing tests with temporary patches to twiddle glance & ironic confs. | |
| 22:27:19 | mordred | efried: woot! | |
| 22:27:45 | efried | morgan mordred sdague And FYI, PS5 has the swizzle where you pass the service_type (rather than the conf_group/project_name) to the util. | |
| 22:27:55 | efried | (which both sdague and morgan asked for) | |
| 22:27:57 | morgan | nice | |
| 22:28:17 | efried | Next stop: cinder! | |
| 22:29:27 | mriedem | huh, we're actually failing grenade at a decent clip http://logs.openstack.org/54/489754/3/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/7e19ce3/logs/old/screen-n-cond.txt.gz#_2017-08-01_21_57_19_559 | |
| 22:29:38 | mriedem | on the ocata side | |
| 22:33:24 | openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | |
| 22:35:20 | cdent | mriedem: in muiltinode does one node stay ocata? is this before or after that? | |
| 22:36:02 | mordred | efried: a few suggestions inline | |
| 22:37:26 | mriedem | cdent: depends on the job config | |
| 22:37:39 | mriedem | some keep n-cpu behind | |
| 22:38:19 | mriedem | but yeah i think multinode grenade always keeps n-cpu behind as n-1 | |