| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-18 | |||
| 14:01:00 | lyarwood | sean-k-mooney: as above, when we ERROR out during scheduling attached volumes remain in a reserved state (as we don't delete their attachments in cinder), if we fail during spawn the compute layer moves the volume back to available (by deleting the attachments in cinder). | |
| 14:01:20 | nightmare_unreal | hello where can I see list of bugs for openstack CLI client or nova related bugs | |
| 14:01:43 | sean-k-mooney | lyarwood: ah ok | |
| 14:01:48 | lyarwood | sean-k-mooney: so I'm thinking that for the latter we should keep the volume reserved to avoid something or someone else coming along and using it | |
| 14:01:58 | lyarwood | if it isn't multiattach etc etc | |
| 14:02:51 | sean-k-mooney | lyarwood: the later being during spawn when we delete the attachemtn we leave it in a reserved sate | |
| 14:03:20 | lyarwood | sean-k-mooney: when we delete the attachments the volume moves back to an available state | |
| 14:03:37 | lyarwood | sean-k-mooney: I'd like to change this to keep it reserved by not deleting the attachments | |
| 14:04:23 | sean-k-mooney | in the case wehre start fails or boot fails or both | |
| 14:04:42 | sean-k-mooney | for boot it will end up in cell 0 and have host so no attachemts | |
| 14:04:57 | sean-k-mooney | *no host | |
| 14:05:17 | lyarwood | for spawn, so we have a host and a cell already | |
| 14:06:29 | lyarwood | brb | |
| 14:24:31 | sean-k-mooney | nightmare_unreal: nova bugs are here https://bugs.launchpad.net/nova | |
| 14:24:45 | sean-k-mooney | nightmare_unreal: the openstack client uses storyborad | |
| 14:25:52 | sean-k-mooney | so here https://storyboard.openstack.org/#!/project/openstack/python-openstackclient but this is all the releated projects https://storyboard.openstack.org/#!/project_group/80 | |
| 14:26:15 | nightmare_unreal | thanks :) | |
| 14:34:45 | dansmith | artom: your sigh about migration func testing is lack of review I assume? | |
| 14:34:55 | dansmith | I'm sure I'm at least partially to blame there | |
| 14:35:51 | artom | dansmith, no, mostly towards myself | |
| 14:35:57 | dansmith | oh okay | |
| 14:35:58 | artom | For not staying on top of it | |
| 14:36:33 | artom | I wanted to do a push, well, starting now | |
| 14:36:51 | artom | I still have stephenfin's feedback from the previous millenium to address (about the 2/3 NUMA nodes split) | |
| 14:37:16 | artom | But if we manage to land it before the VPMEM live migration patch, it would make things easier | |
| 14:37:25 | dansmith | ack | |
| 14:37:48 | dansmith | I'm pretty un-jazzed about that patch | |
| 14:38:24 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support unshelve with qos ports https://review.opendev.org/704759 | |
| 14:38:24 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enable unshelve with qos ports https://review.opendev.org/705475 | |
| 14:38:37 | artom | dansmith, live migration with VPMEM? | |
| 14:38:44 | dansmith | yes | |
| 14:39:10 | artom | In terms of current implementation or the concept in general? | |
| 14:39:34 | artom | I'm just neutral-jazzed - I got involved because my pawprints are all over that code after NUMA live migration | |
| 14:39:56 | dansmith | well, I'm rather negative on the vpmem stuff in general, but that ship has sailed..But yeah, the current patch more acutely | |
| 14:40:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Merge qos related renos for Ussuri https://review.opendev.org/706766 | |
| 14:40:17 | dansmith | it smacks of "let me just move around all these critical things so they work the way I need them" | |
| 14:40:44 | artom | Yeah... | |
| 14:40:49 | lyarwood | sean-k-mooney: https://bugs.launchpad.net/neutron/+bug/1867936 seen this before? | |
| 14:40:49 | openstack | Launchpad bug 1867936 in neutron "test_update_delete_extra_route failing due to timeout when creating subnets" [Undecided,New] | |
| 14:41:28 | artom | dansmith, you've essentially put words on a vague feeling I had | |
| 14:41:43 | sean-k-mooney | lyarwood: only in the failure of one of your patches | |
| 14:41:48 | artom | Which is why I would really be more comfortable if we at least had NUMA LM func tests in-tree before merging | |
| 14:42:07 | gibi | dansmith: hi! added the global service version check to https://review.opendev.org/#/c/705475 | |
| 14:42:08 | artom | Those at least exercise those code paths | |
| 14:42:18 | dansmith | artom: well, I certainly don't think that's unreasonable, as long as we can get that moving | |
| 14:42:33 | artom | dansmith, ack, lemme get on that | |
| 14:42:56 | lyarwood | sean-k-mooney: yeah I've seen it a few times, was just wondering if someones already raised a more specific bug report for this | |
| 14:43:04 | dansmith | artom: using your patch to validate what we have in tree so we can more directly assess the changes this one makes would be helpful | |
| 14:43:07 | dansmith | gibi: ack | |
| 14:43:39 | artom | dansmith, the bottom two of https://review.opendev.org/#/c/705545/ would be ready, I'd say | |
| 14:43:41 | sean-k-mooney | lyarwood: there have been some issue with the dhcp agent by the way. e.g. it not seting up the dhcp config in a timely manner | |
| 14:43:52 | dansmith | gibi: I feel like we might be at the point with the cyborg set where someone else needs to take over it.. I'm not fully caught up yet this morning but I think feedback has been pending for a while now | |
| 14:43:58 | sean-k-mooney | this might just be a side effect of that althogh i though that neuton had fixed that | |
| 14:44:07 | artom | dansmith, if you have any -1s to get in on those, do it now ;) | |
| 14:44:28 | lyarwood | sean-k-mooney: kk | |
| 14:44:29 | dansmith | artom: at least the bottom one is -W, should it not be? | |
| 14:44:42 | gibi | dansmith: I agree. I can spend some review time on that series but I cannot promise to take over the impl | |
| 14:44:51 | artom | dansmith, it's to indicate that the top 2 aren't ready | |
| 14:45:06 | artom | It shouldn't stop you from -1'ing :) | |
| 14:45:13 | dansmith | gibi: no, not asking the latter, but it would be good if we could have the next few above your +2s reviewed and then make a big effort there | |
| 14:45:51 | gibi | dansmith: OK. I will look into those patches soon | |
| 14:46:26 | dansmith | gibi: thanks, I'll start in on your unshelve patch | |
| 14:46:36 | gibi | dansmith: thanks | |
| 14:47:52 | gibi | dansmith: I'm working on retrofitting the service version check to the other qos move operations implemented in Ussuri | |
| 14:48:06 | gibi | dansmith: but the pattern is there in the unshelve patch | |
| 14:48:46 | dansmith | gibi: okay, things that are already landed are kinda already out of the bag, unless you mean gating everything on this service version for people that haven't rolled yet | |
| 14:49:12 | gibi | dansmith: I'm planning to gate it on the service version I added at the unshelve patch | |
| 14:49:17 | gibi | dansmith: does it make sense to you? | |
| 14:49:37 | gibi | dansmith: I don't think many people rolled Ussuri yet | |
| 14:49:52 | dansmith | okay.. technically that is kinda making history go backwards, but for the practical case where people are just deploying releases, it makes sense | |
| 14:50:15 | dansmith | I don't really want to get too far into that line of thinking, because it leads us towards "break things during the release as long as we fix before GA" but... this isn't quite that | |
| 14:50:59 | gibi | dansmith: I consider it as a bug that was introduced in Ussuri and fixed in Ussuri | |
| 14:51:35 | nightmare_unreal | how can i report a bug , it's related to openstack client | |
| 14:51:51 | nightmare_unreal | use storyboard ? | |
| 14:53:54 | stephenfin | nightmare_unreal: yeah, I think so | |
| 15:03:55 | lyarwood | sean-k-mooney: https://zuul.opendev.org/t/openstack/build/b62f4ef603904e05bc34e2a32a4d2866/log/logs/screen-n-cpu.txt#6754 - so the nova-live-migration evacuation failure looks like another race between libvirtd and n-cpu btw | |
| 15:04:00 | lyarwood | at least in this example | |
| 15:08:24 | lyarwood | ah ignore me that's an earlier negative test | |
| 15:15:54 | dansmith | gibi: maybe you can save me some time checking on this, but... it looks like we do set reqspec.requested_resources in various places, | |
| 15:16:09 | dansmith | but this patch just blindly sets that on reqspec (as you do elsewhere) | |
| 15:16:42 | dansmith | is the _only_ time any of requested_resources gets used is for port qos right now and you're just taking the shortcut of being aggressive there? | |
| 15:17:01 | dansmith | I note the comment to that effect, but I just want to make sure you're positive that it's only used for that, and that it's always safe to blow those away | |
| 15:17:34 | gibi | dansmith: I'm positive it is use only for that (except the cyborg case where I'm not sure what is merged yet) | |
| 15:17:50 | dansmith | nothing for it is merged, so that's safe | |
| 15:18:05 | gibi | but I do remember that the cyborg change was proper and did not overwrite the requested_resources | |
| 15:18:37 | gibi | dansmith: so I think we are safe there | |
| 15:18:43 | dansmith | seems a little risky to just be blind about these early cases, and that it's going to be a source of bugs later where cyborg uses it for spawn, for example but unshelve overwrites those | |
| 15:19:13 | openstackgerrit | Lee Yarwood proposed openstack/nova master: gate: Ensure n-cpu is stopped on the subnode during evacuation testing https://review.opendev.org/713674 | |
| 15:19:14 | lyarwood | sean-k-mooney: ^ so I wonder if n-cpu running on the subnode is borking things here even if it is marked as down in the API? | |
| 15:20:04 | lyarwood | I can see the subnode refreshing the instance network info cache for example | |
| 15:20:16 | lyarwood | while we are trying to evacuate | |
| 15:20:17 | gibi | dansmith: the field is not persisted so I don't think we will overwrite existing data from the past. every user of the requested_resources needs to re-fill it for each operation | |
| 15:20:31 | dansmith | ahh, okay that makes sense | |
| 15:25:50 | sean-k-mooney | lyarwood: well the mark host down api has a prequisite that the host is actully down and the agents are stoped | |
| 15:26:39 | sean-k-mooney | lyarwood: if we voilate that expectation then perhaps it coudl cause issues | |
| 15:27:41 | sean-k-mooney | lyarwood: in the evacuate case we proably should be storping n-cpu on the compute host as it is not properly fenced if it is running and able to connect to the message bus/db | |
| 15:28:10 | lyarwood | sean-k-mooney: yup, if it's pinging neutron behind our backs I'm assuming that's going to cause issues when it comes to evacuting the instance from the node | |
| 15:29:09 | sean-k-mooney | well i have not check how the info cache works but unless its per host then it could replace the binding info in the cache with the stale prot bindin info for the orginal host | |
| 15:29:33 | sean-k-mooney | although im not sure if that would cause an issue | |
| 15:29:50 | sean-k-mooney | its certenly safter to not have it messing with the info cache | |