| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-03-22 | |||
| 16:03:23 | sean-k-mooney | tl;dr version | |
| 16:03:45 | sean-k-mooney | what i saw is that in the scudler i saw the log statements showing the code was runnign in the numa filter | |
| 16:04:13 | sean-k-mooney | adn i could see that the network request and numa info was correcty provide by the instace and the resouce tracker | |
| 16:04:22 | sean-k-mooney | so everything at the schduler level is working properly | |
| 16:04:29 | sean-k-mooney | fwhat i observed on the compute node | |
| 16:04:46 | sean-k-mooney | is that none of the log stames for the numa aware vswithc code paths were printing | |
| 16:05:11 | sean-k-mooney | so even thouhg both the filter and compute agaisnt call the same funtions in hardware.py | |
| 16:05:24 | sean-k-mooney | for some reasons the compute agent was not running that code | |
| 16:05:41 | artom | sean-k-mooney, https://meet.google.com/fhz-qcxz-rsy | |
| 16:05:54 | artom | (We can have that URL public, right?) | |
| 16:05:57 | sean-k-mooney | that implies that there is an issuee with the numna toplogy reqest we are generating/passing in the compute agent to that common code | |
| 16:06:00 | artom | I mean too late now | |
| 16:06:13 | sean-k-mooney | sure anyone is welcome | |
| 16:06:37 | sean-k-mooney | its i just wanted something higher bandwiht | |
| 16:13:10 | opendevreview | ribaudr proposed openstack/nova-specs master: Update and Re-propose "Allow Manila shares to be directly attached to an instance when using libvirt" for Dalmatian https://review.opendev.org/c/openstack/nova-specs/+/913997 | |
| 17:59:41 | opendevreview | Merged openstack/nova master: Update min support for Dalmatian https://review.opendev.org/c/openstack/nova/+/913890 | |
| 18:27:27 | opendevreview | Merged openstack/nova stable/2023.2: pwr mgmt: handle live migrations correctly https://review.opendev.org/c/openstack/nova/+/913198 | |
| #openstack-nova - 2024-03-23 | |||
| 12:09:53 | opendevreview | norman shen proposed openstack/nova master: Remove an unncessary LOG https://review.opendev.org/c/openstack/nova/+/912661 | |
| #openstack-nova - 2024-03-25 | |||
| 08:17:43 | opendevreview | Sylvain Bauza proposed openstack/nova master: Update contributor guide for 2024.2 Dalmatian https://review.opendev.org/c/openstack/nova/+/913481 | |
| 08:30:08 | opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: AggregateMultitenancyIsolation to support unlimited tenant https://review.opendev.org/c/openstack/nova/+/896512 | |
| 08:30:27 | sahid | bauzas: o/ any chance to make progress on that one? | |
| 08:31:20 | bauzas | sahid: good point, thanks for your resilience :) | |
| 08:32:19 | bauzas | sahid: I'm adding the blueprint to tomorrow's nova meeting for asking to be a specless one | |
| 08:34:22 | bauzas | (added now https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting ) | |
| 09:28:35 | sahid | bauzas: thanks a lot :-) yes waiting for a while now, we have backported the patch anywy in our production but it may be useful for other users as it address a legit use-case | |
| 11:42:52 | sean-k-mooney[m] | bauzas: while i orginally was not in favor of sahid scheduler enhancement it became apparent that we would need to enchance palcement to do it any other way and that would be non trivial so im ok with proceeding with it as a specless blueprint again | |
| 11:51:18 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add missing Controller inheritance https://review.opendev.org/c/openstack/nova/+/914083 | |
| 11:51:18 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Remove FlavorManageController https://review.opendev.org/c/openstack/nova/+/914084 | |
| 12:53:13 | sean-k-mooney | bauzas: fyi this was my fun friday rabbit hole https://bugs.launchpad.net/nova/+bug/2058928 | |
| 12:53:55 | sean-k-mooney | the title says it all but we shoudl proably fix that | |
| 13:30:13 | bauzas | sean-k-mooney: looking | |
| 13:31:00 | artom | sean-k-mooney, btw, I couldn't reproduce that vswitch bug... our internal BZ isn't great on steps to reproduce, but do you recall doing anything specific? | |
| 13:31:12 | bauzas | sean-k-mooney: so we're missing instance actions, right? | |
| 13:31:46 | sean-k-mooney | bauzas: yep we never recored any in the compute manage to say if the attach/detach finished with or without an error | |
| 13:31:57 | sean-k-mooney | bauzas: so we get an instance action record with no events | |
| 13:32:35 | sean-k-mooney | i tried to make tempest use teh compeltion of the instance ation by polling and it timed out because we never actully create the finish event | |
| 13:32:55 | sean-k-mooney | we do emit the notificaitons we just dont save the event compeltion/failure in the db | |
| 13:33:34 | sean-k-mooney | we already have the remotable methods to do that including a handy class modthod on the instance_action object | |
| 13:33:43 | sean-k-mooney | we just dont call it | |
| 13:34:20 | sean-k-mooney | artom: i create a two node devstack and manually repoduced it | |
| 13:34:24 | bauzas | okay, so I guess you created a bug report for making sure we could backport it then ? | |
| 13:34:31 | bauzas | but I'm afraid of the API so | |
| 13:34:41 | bauzas | could be a interop issue | |
| 13:34:44 | sean-k-mooney | i created the bug to not loose track of this | |
| 13:35:00 | sean-k-mooney | i dont think it would be an interop thing but i wanted to discsss this at the ptg | |
| 13:35:13 | bauzas | okay, I'm asking that because I don't know whether we could rather a blueprint instead of a bug | |
| 13:35:18 | sean-k-mooney | or get gmann's input | |
| 13:35:22 | sean-k-mooney | bauzas: its not a blueprint | |
| 13:35:24 | bauzas | sean-k-mooney: I created a new etherpad | |
| 13:35:46 | sean-k-mooney | bauzas: we dont actully need to change the api | |
| 13:35:48 | bauzas | sean-k-mooney: could you add it there ? https://etherpad.opendev.org/p/nova-dalmatian-ptg | |
| 13:36:02 | bauzas | sean-k-mooney: no, I'm not saying it would change the API | |
| 13:36:02 | sean-k-mooney | there is a filed called events in teh the action record | |
| 13:36:12 | sean-k-mooney | its currently empty for at least some of the instance actions | |
| 13:36:18 | sean-k-mooney | but apprenly stop populates it | |
| 13:36:30 | sean-k-mooney | so we would jsut be populating that with event for the rest | |
| 13:36:36 | bauzas | but it would return different actions between different API versions | |
| 13:36:40 | sean-k-mooney | that said we may or may not want a microversion to signal that | |
| 13:36:47 | sean-k-mooney | no | |
| 13:36:50 | sean-k-mooney | the actions woudl not change | |
| 13:36:55 | bauzas | (ie. Caracal wouldn't return some action while Dalmatian would) | |
| 13:37:09 | sean-k-mooney | the number of actions woudl be identiacal | |
| 13:37:15 | sean-k-mooney | the payload of the action woudl change | |
| 13:37:20 | sean-k-mooney | one sec ill get the example | |
| 13:37:35 | artom | sean-k-mooney, I believe you - I guess I'm asking if it was 100% reproducible, and did you need to do anything else besides the hw:numa_nodes=1 flavor extra spec to trigger a NUMA topology | |
| 13:37:50 | sean-k-mooney | bauzas: https://docs.openstack.org/api-ref/compute/#show-server-action-details | |
| 13:38:04 | bauzas | I mean the action would have a finish event https://docs.openstack.org/api-ref/compute/#show-server-action-details | |
| 13:38:34 | bauzas | so I don't know if this is an interop issue | |
| 13:38:34 | sean-k-mooney | yes within the events list in the action | |
| 13:38:53 | bauzas | but yeah we can discuss this with gmann sure | |
| 13:39:02 | bauzas | sean-k-mooney: can you add this bug to the PTG etherpad ? | |
| 13:39:03 | sean-k-mooney | its purly additive, populating an existing filed | |
| 13:39:07 | sean-k-mooney | bauzas: sure | |
| 13:39:14 | bauzas | thanks | |
| 13:39:43 | sean-k-mooney | artom: ya pretty much. not 100% but enough thaat it happend on the first try and several in a row | |
| 13:39:48 | bauzas | sean-k-mooney: yeah, my concern is that users would know which release is used for an API just by calling instance actions list :) | |
| 13:40:09 | bauzas | if you can find a finish event, then it's a Dalmatian version | |
| 13:40:36 | sean-k-mooney | yes although that would only matter if we didnt have a microversion fro dalamtion | |
| 13:40:50 | sean-k-mooney | and since we hope to have at least 1 for the maniala shares | |
| 13:40:55 | sean-k-mooney | im not sure this is a concern | |
| 13:41:16 | bauzas | well, this is not a concern from me, rather an open question | |
| 13:41:19 | artom | sean-k-mooney, so, NUMA placement is deterministic, right? (We don't promise it, but de facto it is)... So if this is not happening 100% of the time, it means there's a race or something similarly *non-deterministic* | |
| 13:41:25 | sean-k-mooney | personally i think this is somethign that shoudl be backported once fixed but again we can caht about that | |
| 13:41:41 | bauzas | I wonder #1 whether we need a microversion #2 whether we can backport it | |
| 13:41:48 | artom | When you say "reproduced", you mean the logs for the fitting of network metadata not appearing? | |
| 13:42:03 | bauzas | what I'd prefer is that #1 no and #yes :) | |
| 13:42:05 | sean-k-mooney | artom: deteministic in the sense that if you had only one host and you started with it empty and booted vms in the same sequence you would expect the same result yes | |
| 13:42:32 | sean-k-mooney | artom: obviouly in reality you have mutiple concurrent requests | |
| 13:42:33 | bauzas | sorry, meant : MHO : #1 no and #2 yes (no microversion, and yeah we can backport) | |
| 13:42:42 | sean-k-mooney | so you cant really prediect it 100% | |
| 13:42:47 | bauzas | but let's ask gmann and other folks | |
| 13:42:54 | sean-k-mooney | artom: but for functioanl tests it shoudl be prediable | |
| 13:43:10 | artom | sean-k-mooney, but in the context of a functional test with one host and one (or two) instances, it's fully predictable | |
| 13:43:32 | sean-k-mooney | artom: it should be yes | |
| 13:43:56 | sean-k-mooney | give me a minute to finish up a few more things and if you want we can try pair programming a repoducer in a bit | |
| 13:44:11 | artom | So if I'm not reproducing it (barring the possibility of me doing something wrong with my test), it means the cause is something non-deterministic... | |
| 13:44:32 | sean-k-mooney | artom: no it could mean our test fixtures are maskign it | |
| 13:45:36 | sean-k-mooney | on firday i tought i knew how to repoduce it but i need to recall what that was. | |
| 15:06:08 | dansmith | mnaser: do you have any sort of preprod environment you could test the eventlet hub switch-to-asyncio for us? | |
| 15:06:34 | dansmith | I'm worried about performance and stability issues by just blindly making that change, and I don't think a small-node-count devstack tempest run is good enough to say it's reasonable | |