Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-03-02
17:30:23 Sundar Not 'we feel like it;, but whether it is tested and whether there are grounds to believe it will work dafely all the time
17:30:26 sean-k-mooney we agreed not to block the ops and just document them
17:30:28 Sundar *safely
17:30:41 sean-k-mooney that is why https://review.opendev.org/#/c/674726 was abandoned
17:30:55 Sundar Hmm, I recall otherwise. Let me look for past IRC discussions.
17:31:39 sean-k-mooney we can bring it back but i dont think its ok to block things like rebuild or volume based snapthot when we know they should work and they do work at least with the fake driver
17:39:22 Sundar sean-k-mooney: Here is our past discussion: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2020-02-07.log.html#t2020-02-07T19:17:13 . I see different opinions, but you still want the doc-only approach. But you do agree that people won't read the docs till things break
17:39:23 Sundar (http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2020-02-07.log.html#t2020-02-07T19:31:39)
17:39:54 Sundar Is it not safer to block an unproven op (but only for accelerators obviously)?
17:40:09 Sundar I am not si sure that snapshots will work
17:40:11 sean-k-mooney that is form last month not form when we deiced not to block everything before christmas
17:40:24 sean-k-mooney a live snapshot will work
17:40:41 sean-k-mooney a cold snapshot might should too
17:40:53 sean-k-mooney since it stops the vm and then starts it after
17:41:37 Sundar A snapshot is going to save the mmap'd buffers in VM memory, and bring it back on restore. But the accelerator state would have changed in between as the VM continued op. So, on restore, the buffers in the VM will not match the accelerator contents.
17:42:07 sean-k-mooney Sundar: no it wont
17:42:21 sean-k-mooney a snappshot only makes a copy of the root disk
17:42:38 sean-k-mooney its not the same as a vm snapshot in virtual box or hyperv
17:42:44 Sundar I thought there's an option to do snapshot with RAM.
17:42:51 sean-k-mooney no
17:42:57 sean-k-mooney not in the libvirt dirver
17:43:10 sean-k-mooney you can do that with qemu but not via nova
17:43:13 Sundar libvirt allows it -- but apparently not the Nova libvirt driver?
17:43:27 Sundar I see
17:43:38 sean-k-mooney yes becasue we have no where to store the ram dump
17:44:31 sean-k-mooney we could in theory upload it as a second image to glace or something but snapshot in the nova api is just of the disk
17:46:00 Sundar Are you able to check snapshot with FPGA-attached VMs? I'll try too -- but what works in the lab may not always work in the field.
17:46:41 sean-k-mooney i have not deployed on real hardware so no.
17:48:40 Sundar I would say that it is safer to block something rather than speculatively allow it. Do you have users who strongly need snapshots with accelerators?
17:49:43 sean-k-mooney not specifically but i would not consider movingit form techpreview without it
17:50:10 sean-k-mooney in the curent patch there are no ways to do any move operation and you cant resuceu or snapshot the vm
17:50:28 sean-k-mooney so there is 0 way to do maintaince on the host or recover in the case of a vailure
17:50:33 sean-k-mooney even evacuate is blocked
17:50:58 sean-k-mooney as is adding or removeing volumes or interfaces
17:51:22 sean-k-mooney so with only the abilty to boot and delete a vm we would not support this in productin for customer to use
17:52:21 sean-k-mooney thats a downstrema desciion however. it wont affect if we support this in upstream
17:52:56 Sundar I see your concern. The real solution is to test these ops well with real accelerators and real-ish workloads (not sample bitstreams). I'll pass that on to my Cyborg/Intel colleagues.
17:53:22 Sundar Meanwhile, based on your input, I could remove the blocks on snapshot and rescue. Presumably backup too?
17:54:16 sean-k-mooney backup is basicaly a cron job in the nova code that calls snapshot and clean up old snapshots
17:54:27 sean-k-mooney so if snapshot work then back will
17:55:24 sean-k-mooney im not sure about rescue. it should work but i dont know if the acclerator will be aviable while its in resuce state
17:55:53 sean-k-mooney if the accelerator in not present in the vm during or after resuce has been completed then we might need to block it
17:56:32 Sundar Since the rescue brings in a different disk image, which may not have the right device drivers, it probably will not. The unrescue to the previous image "should" work.
17:57:04 sean-k-mooney oh the other hand not everyone would need the acclerator to be present during resuce so as long as we dont loose itafter unrescue we should be ok
17:57:23 Sundar Ok. So rescue is in. What about suspend? It doesn't work today because we don't have the code to handle accelerators in that path.
17:57:51 sean-k-mooney suspend wont work without a code change so ya that can remain blocked
17:58:31 openstackgerrit Mark Goddard proposed openstack/nova master: Fix PCI passthrough race on reschedule (claims) https://review.opendev.org/710847
17:58:32 openstackgerrit Mark Goddard proposed openstack/nova master: Fix PCI passthrough race on reschedule (refresh) https://review.opendev.org/710848
17:59:23 Sundar sean-k-mooney: The other 2 ops mentioned in https://review.opendev.org/#/c/674726/3/nova/tests/functional/test_servers.py are rebuild and resize.
17:59:42 dansmith Sundar: sean-k-mooney just catching up, but if we can't support snapshot, that's a huge problem IMHO (although I like sean-k-mooney think it's likely not a problem)
18:00:19 dansmith if rescue doesn't include the accelerator I don't think we should block it
18:00:33 dansmith rescue is a special environment
18:00:46 dansmith I don't really know why it wouldn't include it, but I don't think that's a major problem or anything
18:01:02 sean-k-mooney ya i think rescue should be fine just as long as we validate the acclertor is present after unrescue when we go back to the stanard env
18:01:21 dansmith yes, that's obviously an important point :D
18:01:34 sean-k-mooney Sundar: resize wont work
18:01:49 sean-k-mooney i tested that with the fake dirver and it does nto update placmenet correctly
18:01:57 sean-k-mooney rebuild should work fine
18:04:01 sean-k-mooney Sundar: dansmith i left comment on https://review.opendev.org/#/c/674726 with what i think should and should not work
18:10:23 sean-k-mooney Sundar: by the way im going to be AFK or on PTO for most of the rest of this week so if i dont repond or am offline that is why
18:17:38 Sundar sean-k-mooney: dansmith: I'll respond in the patch. Meanwhile I have filed a Cyborg storyboard task: https://storyboard.openstack.org/#!/story/2007358 . I'll raise that with Cyborg folks.
18:18:34 sean-k-mooney Sundar: is there a thirdpart hardwar ci somewhere?
18:18:54 sean-k-mooney if so they could just do what im doing in the multinode job
18:19:15 sean-k-mooney that will validate all the lifecycle ops
18:19:55 sean-k-mooney they could blacklist the tests they know do not work and as the missing ops get adressed they can be removed form the blacklist
18:20:19 Sundar sean-k-mooney: Unfortunately, our current plans for 3pCI with FPGAs are nixed. Our colleagues in other parts of Intel could possibly resume it, since we have the hardware lined up.
18:20:24 sean-k-mooney alternitvly the cyborg plugin could be extended but that really up to whowever is going to test it
18:20:47 sean-k-mooney ok ya i kind of guessed that would be the case
18:22:25 Sundar sean-k-mooney: Noted your PTO. Thanks for your help. I'm not sure how long I'll be around after this week.
18:34:29 sean-k-mooney ok time for dinner.
18:34:43 sean-k-mooney o/
19:33:25 openstackgerrit Merged openstack/nova master: Fix os-attach-interfaces policy to be admin_or_owner https://review.opendev.org/705135
19:50:53 openstackgerrit David Hill proposed openstack/nova stable/queens: Port binding based on events during live migration https://review.opendev.org/710861
19:51:09 openstackgerrit David Hill proposed openstack/nova stable/queens: Port binding based on events during live migration https://review.opendev.org/710861
20:21:51 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Use virDomainBlockCopy to swap volumes with >= 5.10.0 https://review.opendev.org/696834
20:21:52 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP nova-next: Start testing the q35 machine type https://review.opendev.org/708701
20:30:21 openstackgerrit Merged openstack/nova master: Add test coverage of existing os-aggregates policies https://review.opendev.org/701651
20:48:26 openstackgerrit Merged openstack/nova master: Add test coverage of existing os-assisted_volume_snapshots policies https://review.opendev.org/701835
21:37:56 redrobot Hi Nova friends! I'd love to get another set of eyes on https://review.opendev.org/#/c/680562/
21:58:53 mnaser what's the best way of forcing an instance type to specifically boot on a specific host now? if i use the filter which uses an aggregate's instance_type, i would have to put all the others in an aggregate with instance_type including all the other instance types
21:59:52 mnaser the only issue is that in this case, i have no simple way of enforcing no other instance type than $foo lands on this vm
22:01:03 mnaser if i use traits, i have to use forbidden/required but that means if someone creates a flavor and forgets to add the forbidden, it'll end up on that host possibly
22:08:32 dansmith mnaser: does this help?
22:08:33 dansmith https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html
22:08:39 dansmith that was the point of that work, IIRC
22:10:37 mnaser dansmith: ah that's perfect, sorry i tried looking around but did not find that
22:10:47 dansmith it's recent
22:13:56 mnaser neat, thank you.
#openstack-nova - 2020-03-03
00:00:31 openstackgerrit Brin Zhang proposed openstack/nova master: Fix os-volumes-attachments policy to be admin_or_owner https://review.opendev.org/709955
00:03:38 joshualyle does anyone have any ideas on why my compute nodes would only seem to schedule migrations between subsets of nodes?
00:04:10 joshualyle I upgraded half of my nodes and then the other half but now the latter half will only live-migrate between themselves and I'm not sure why
00:08:04 openstackgerrit Brin Zhang proposed openstack/nova master: Add test coverage of existing os-volumes-attachments policies https://review.opendev.org/709929
00:08:21 openstackgerrit Brin Zhang proposed openstack/nova master: Introduce scope_types in os-volumes-attachments policy https://review.opendev.org/709388
00:16:59 openstackgerrit Merged openstack/nova master: Add new default roles in Admin Action API policies https://review.opendev.org/676682
00:19:53 openstackgerrit Brin Zhang proposed openstack/nova master: Add test coverage of existing os-volumes-attachments policies https://review.opendev.org/709929
00:21:03 openstackgerrit Brin Zhang proposed openstack/nova master: Introduce scope_types in os-volumes-attachments policy https://review.opendev.org/709388
00:23:14 openstackgerrit Merged openstack/nova master: Introduce scope_types in os-assisted_volume_snapshots policy https://review.opendev.org/701837
00:23:21 openstackgerrit Merged openstack/nova master: Add new default roles in os-assisted_volume_snapshots policies https://review.opendev.org/701840
00:23:32 openstackgerrit Merged openstack/nova master: api-ref: mark device response param as optional for list/show vol attachments https://review.opendev.org/690383
00:25:57 openstackgerrit Brin Zhang proposed openstack/nova master: Add new default roles in os-volumes-attachments policies https://review.opendev.org/710190

Earlier   Later