| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-19 | |||
| 11:39:44 | sean-k-mooney | mnaser: cool i hope the placement work is finished in tripleo by now | |
| 12:05:15 | bauzas | mmm, super weirdo : http://paste.openstack.org/show/777697/ | |
| 12:05:29 | bauzas | but https://docs.openstack.org/api-ref/compute/?expanded=evacuate-server-evacuate-action-detail,update-forced-down-detail#update-forced-down tells me it's right | |
| 12:05:46 | bauzas | gibi: any ideas ? ^ | |
| 12:05:53 | bauzas | or sean-k-mooney: ^ | |
| 12:06:53 | bauzas | 'host' and 'binary' are correct arguments for force-down (at least for 2.77 microversion) | |
| 12:07:30 | bauzas | I don't see why I'm getting a HTTP400 | |
| 12:08:07 | gibi | looking | |
| 12:08:32 | bauzas | thanks | |
| 12:08:58 | gibi | Starting with microversion 2.53 this API is superseded by PUT /os-services/{service_id}. | |
| 12:09:39 | bauzas | ah | |
| 12:09:58 | bauzas | I remember that but the API docs don't tell it AFAICS | |
| 12:10:17 | gibi | s | |
| 12:10:17 | bauzas | holy shit, it does | |
| 12:10:17 | gibi | I copied this note from the API doc | |
| 12:10:23 | bauzas | yeah, me facepalms | |
| 12:10:49 | bauzas | FWIW, I just used some existing helper method, but I guess it sticks with an old microversion call then | |
| 12:11:02 | gibi | yeah, could be | |
| 12:11:16 | bauzas | it does indeed, 2.14 | |
| 12:11:28 | bauzas | we should deprecate this caller | |
| 12:11:59 | bauzas | and tell to only use with microversion less than 2.53 (or force an older microversion) | |
| 12:15:32 | sean-k-mooney | huh so that is what that looks like when you uses the wrong microversion | |
| 12:17:41 | bauzas | indeed, but I had to RTFM | |
| 12:17:55 | bauzas | (which I did, but my pink eyes got me wrong apparently) | |
| 12:18:10 | bauzas | so, nothing to change from an API perspective IHM | |
| 12:18:12 | bauzas | IMHO | |
| 12:18:34 | bauzas | it's just the helper method which would ideally add some docstring | |
| 13:12:43 | openstackgerrit | Merged openstack/nova master: make config drives sticky bug 1835822 https://review.opendev.org/669738 | |
| 13:12:44 | openstack | bug 1835822 in OpenStack Compute (nova) "vms loose acess to config drive with CONF.force_config_drive=True after hard reboot" [Medium,In progress] https://launchpad.net/bugs/1835822 - Assigned to sean mooney (sean-k-mooney) | |
| 13:13:34 | sean-k-mooney | TIL things can merge on there 1st time in the gate queue | |
| 13:13:41 | sean-k-mooney | is suspcious | |
| 13:22:37 | mordred | sean-k-mooney: yeah. my first thought when something is green the first time is that I did something wrong | |
| 13:23:36 | sean-k-mooney | i think we are finally getting past the gate instablity and have nailed down some of the intermitly failing tests | |
| 13:24:03 | sean-k-mooney | there are still more work to be done but its looking a lot healtier then it was on monday | |
| 13:29:18 | mordred | \o. | |
| 13:29:20 | mordred | gah | |
| 13:29:22 | mordred | I meant | |
| 13:29:24 | mordred | \o/ | |
| 13:29:28 | mordred | stupid keyboards and fingers | |
| 13:30:27 | bauzas | mordred: just use a French keyboard for a minute and you'll love the current keyboard you have | |
| 13:30:35 | bauzas | Alt Gr, my love | |
| 13:33:56 | sean-k-mooney | i use one of these with cherry mx brown switches. a mechanical keyboard make typing better regardless of the layout | |
| 13:33:59 | sean-k-mooney | https://www.corsair.com/eu/en/k70-rgb-gaming-keyboard | |
| 13:41:50 | bauzas | mriedem: around ? | |
| 13:42:20 | bauzas | mriedem: I saw your comment on https://review.opendev.org/#/c/670112/ | |
| 13:42:25 | bauzas | mriedem: thanks for it | |
| 13:42:31 | bauzas | that said, I have one concern | |
| 13:43:00 | bauzas | if we are adding some check that depends on an unfix bug, then we would need to fix this test when the bugfix arrives | |
| 13:43:30 | bauzas | I mean, I'm totally ok with doing an evacuation | |
| 13:44:19 | bauzas | but I just want to make sure that we would still have orphaned allocations even if someone fixes the "undeleted allocations" stuff when a host is deleted | |
| 13:44:20 | mriedem | there are other ways to orphan migration-based allocations, like resizing a server and then before confirming delete the source compute service | |
| 13:44:42 | mriedem | there might be other ways | |
| 13:44:48 | mriedem | i do'nt know when those bugs will be fixed | |
| 13:44:54 | mriedem | my brain is also not really functioning atm | |
| 13:44:58 | bauzas | no worries | |
| 13:45:19 | bauzas | you know what ? I'll just stick with your plan and leave a big fat comment, others could voice | |
| 13:57:03 | efried | Nova meeting in 3 minutens | |
| 13:57:15 | efried | ^ pseudo-german, apparently | |
| 13:59:23 | mriedem | efried: so you're running it? | |
| 13:59:26 | mriedem | i just updated the agenda fwiw | |
| 13:59:39 | efried | oh, thank you, I guess I'll get a merge conflict then | |
| 14:12:10 | bauzas | oh crap, discovered a very interesting case | |
| 14:12:18 | bauzas | https://docs.openstack.org/api-ref/placement/?expanded=delete-allocations-detail#delete-allocations deletes all allocations from this consumer | |
| 14:12:37 | bauzas | but that supposes there are no duplicate allocations having the same consumer UUID between two RPs | |
| 14:12:51 | bauzas | which isn't the case when you evacuate an instance and delete the source host | |
| 14:13:16 | bauzas | we're missing some API syntax to say 'I just want *this* allocation attached to *this* RP | |
| 14:16:23 | bauzas | that's sad because if you run my new audit command, then you'll loose your allocation on an existing instance until you heal it | |
| 14:18:38 | gibi | bauzas: In my mind there is not two allocation with the same consumer UUID but there is a singel consumer that allocates from two RPs. | |
| 14:18:56 | bauzas | gibi: sure, but at the end, we delete both records | |
| 14:19:21 | bauzas | anyway, I'll amend my audit command to be preventive and claim there are duplicate records | |
| 14:19:28 | bauzas | instead of deleting | |
| 14:19:31 | stephenfin | mriedem: fwiw, that live-migrate job failed because placement-api didn't start in time. I'm assuming that couldn't be related to the change, which adds checks on compute-manager startup | |
| 14:19:45 | mriedem | stephenfin: yeah i just left a comment about that | |
| 14:19:47 | mriedem | likely a slow node | |
| 14:19:52 | mriedem | devstack times out at 60 seconds | |
| 14:20:17 | stephenfin | Ah, cool. Thanks for checking | |
| 14:28:09 | mriedem | stephenfin: fwiw it looks rare: | |
| 14:28:12 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%5BERROR%5D%5C%22%20AND%20message%3A%5C%22placement-api%20did%20not%20start%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | |
| 14:33:53 | donnyd | FWIW all three of those ran on FN, so I am curious if its slowness or something else | |
| 14:36:34 | kashyap | mriedem: Dear cupid, just to bounce back on the ping from yesterday, I've responded earlier in the day to mnaser -- http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-09-19.log.html#t2019-09-19T08:25:57 | |
| 14:37:03 | kashyap | mriedem: And we ended up being on the same sentence, same word: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-09-19.log.html#t2019-09-19T11:21:12 | |
| 14:37:07 | kashyap | :D | |
| 14:37:33 | mriedem | donnyd: i'm assuming slowness, placement-api workers took 70+ seconds to start, | |
| 14:37:36 | mriedem | devstack times out at 60 | |
| 14:38:05 | mriedem | kashyap: ack | |
| 14:39:06 | kashyap | I'll also send a patch to libvirt docs to explicitly note that 'host-model' will be "expanded" for the live guest. | |
| 14:41:01 | donnyd | https://grafana.fortnebula.com/d/9MMqh8HWk/openstack-utilization?orgId=2&from=1568848907005&to=1568854899362 | |
| 14:41:25 | donnyd | Here is the load snapshot from one of the failures.. doesn't look like anything is overloaded | |
| 14:47:02 | lyarwood | does anyone recall where we actually call initialize_connection during a LM for the destination when using cinderv2? | |
| 14:47:21 | lyarwood | I was sure it was hidden in pre_live_migration but I can't seem to find it. | |
| 14:53:02 | openstackgerrit | sean mooney proposed openstack/nova-specs master: Add spec for vm scoped sriov numa affinity https://review.opendev.org/683174 | |
| 14:53:26 | sean-k-mooney | efried: so you are saying i should start writing more specs now :) | |
| 14:54:12 | efried | sean-k-mooney: no, I'm saying there's no point writing a spec for something brand new. | |
| 14:54:21 | sean-k-mooney | https://review.opendev.org/#/c/674072/3 is the code for ^ by the way | |
| 14:54:41 | sean-k-mooney | because we have enough of a backlog already to finish | |
| 14:54:57 | mriedem | lyarwood: it is in pre_live_migration from the compute manager | |
| 14:55:09 | mriedem | eventually calls down into nova.virt.block_device - refresh_connection_info method | |
| 14:55:25 | mriedem | lyarwood: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6719 | |
| 14:55:47 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/block_device.py#L677 | |
| 14:55:56 | efried | sean-k-mooney: that, and because it's absolute hell for the "customer" relying on an upstream feature to meet a business commitment to have no better than a 60% chance of the feature actually landing, despite being "approved" (or even *prioritized*) for the release. I'm hoping by reducing scope we can increase that percentage. | |