Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-15
15:57:33 mriedem we don't have any interfaces for deleting compute node records either
15:57:39 mriedem we've talked about adding that to nova-manage in the past
15:58:23 penick artom_ almost. Before becoming bros you have to go through the ceremony of course.
15:58:54 mriedem we have another issue with cleaning up stuff in placement mentioned here https://review.openstack.org/#/c/502306/17/specs/rocky/approved/bandwidth-resource-provider.rst@228
15:59:13 cfriesen_ mriedem: we've got a reported bug that deleting then re-adding a compute node with the same name results in not being able to schedule due to this placement issue
15:59:21 mriedem if you disable a compute or it's down, and delete instances from it and then purge them from the db, the allocations for those instances against that compute are never removed from placement
16:00:01 cfriesen_ ew
16:00:02 mriedem cfriesen_: well, how did they delete the compute node record? directly in the DB?
16:00:13 mriedem if you're monkeying with the db, then you are also going to need to deal with cleaning up placement as a result
16:00:20 mriedem at least for now, until we have some interface for this
16:00:34 cfriesen_ mriedem: I don't think they did delete the comptue node record, just the service
16:01:04 mriedem then i don't see how the compute node uuid would change unless the hostname changed or something
16:01:26 mriedem if the hostname changes, you have other problems
16:02:03 mriedem i think this also might tie into something tssurya was looking at which is having a way to delete host mappings and also make that recursive to delete compute nodes
16:02:08 mriedem and placement entries for those compute nodes
16:03:23 mriedem anyway, there is enough here that we should (1) have a way to cleanup compute nodes and their associated records and (2) that is complicated enough that it probably deserves at least a short spec
16:03:42 mriedem if it were easy it would have been done by now
16:03:59 tssurya I am cleaning up compute node and rp records in my patch
16:04:07 tssurya when we delete a cell or a host
16:04:26 dansmith tssurya: s/host/host mapping/ right?
16:04:55 tssurya yes
16:05:37 mriedem cfriesen_: in the meantime, they can use osc-placement 1.0 to delete the old compute node RP in placement
16:06:44 cfriesen_ okay, thanks for the info. tssurya, can you invite me to review that patch?
16:07:10 mriedem https://review.openstack.org/#/c/542964/ ?
16:07:19 cfriesen_ mriedem: I think that's basically what they were planning on doing for now
16:07:25 tssurya https://review.openstack.org/#/c/546660/
16:08:10 tssurya mriedem: its the one I posted, same one needing the functional test
16:08:13 tssurya for the placement side
16:08:32 tssurya for deleting the RP record
16:09:27 tssurya cfriesen_ : done, I hope I got the name right
16:09:31 tssurya for the review invite
16:10:18 cfriesen_ on a totally different issue, I opened https://bugs.launchpad.net/nova/+bug/1755981 Basically there seems to be a race due to the fact that we remove serial ports from ALLOCATED_PORTS but leave them in the guest definition.
16:10:19 openstack Launchpad bug 1755981 in OpenStack Compute (nova) "powering off and on an instance can result in instance boot failure due to serial port handling race" [Undecided,New]
16:10:22 cfriesen_ tssurya: yes, thanks
16:10:51 mriedem cfriesen_: probably ask sahid_ about that bug
16:10:59 mriedem markus_z would have been good too but he's no longer around
16:14:11 cfriesen sahid_: (or anyone else) regarding the above bug, why do we remove the port from ALLOCATED_PORTS when powering off? wouldn't it make more sense to remove them when undefining or changing the libvirt domain? The alternative seems to be to detach the serial ports from the libvirt domain when doing a power off, which is kind of icky.
16:17:03 sahid_ cfriesen: no comment or todo to explain why? I can't really remember
16:18:00 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: Resource provider examples https://review.openstack.org/553461
16:18:46 sahid_ cfriesen: i just looked at the code, furing power_on we are doing a hard-reboot (don't ask me why)
16:19:03 sahid_ it seems that during hard reboot we redefine the domain xml
16:34:35 artom_ jaypipes, wanna talk about https://review.openstack.org/#/c/552722/1/specs/rocky/approved/live-migration-with-cpu-pinning.rst@16?
16:37:06 openstackgerrit Chris Dent proposed openstack/nova-specs master: Spec for isolating configuration of placement database https://review.openstack.org/552927
16:37:39 cdent stephenfin: addressed your suggestions on ^
16:41:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Unmap compute nodes when deleting host mappings in delete cell operation https://review.openstack.org/542964
16:42:08 mriedem tssurya: would you like to propose a backport to stable/queens for ^ ?
16:43:14 tssurya mriedem: yes I will do it
16:45:24 openstackgerrit Alvaro Lopez Garcia proposed openstack/nova master: Ensure that periodic reclaim cleans DB deleted instances https://review.openstack.org/323250
16:46:13 cfriesen sahid_: I'm back. So on power_off() we _destroy() the instance (but leave it defined) and remove the serial ports from ALLOCATED_PORTS. Then on power_on() we hard reboot the instance, which calls _destroy() again, which removes the ports from ALLOCATED_PORTS again, which might result in removing ports currently in use by another instance.
16:46:42 jaypipes artom_: sure, what's up?
16:47:16 openstackgerrit sahid proposed openstack/nova master: libvirt: slow live-migration to ensure network is ready https://review.openstack.org/497457
16:47:24 cfriesen sahid: I assume we call _hard_reboot() to clean up as many things as possible about the instance (given the comment by mdbooth in _hard_reboot())
16:47:27 openstackgerrit Eric Fried proposed openstack/nova master: ProviderTree.{add|remove}_{traits|aggregates} https://review.openstack.org/553475
16:48:05 efried dansmith, jaypipes, cdent, edleafe: One half of the result of the "need to be able to merge traits/aggs" discussion ^
16:48:55 cfriesen sahid_: I'm wondering whether we should do the serial port removal from ALLOCATED_PORTS in _undefine_domain() instead.
16:49:16 openstackgerrit Eric Fried proposed openstack/nova master: update_provider_tree devref and docstring updates https://review.openstack.org/553476
16:49:31 efried dansmith, jaypipes, cdent, edleafe, mriedem: The other half ^
16:49:52 efried ...and now to send out that dev ML note, so I don't get in trouble with mriedem...
16:51:15 tssurya mriedem: there is a small problem with the backport, I guess the above patch sits on this one -> https://review.openstack.org/#/c/540073/ , so will backport both
16:51:30 artom jaypipes, so, what I think I failed in communicating is that getting NUMA resources modelled in placement and claimed by the scheduler (Sylvain's spec) is a dependency of my spec
16:51:53 artom jaypipes, and what I *think* you're saying is that, we can continue using the current compute-node-claims-the-resources way for now
16:53:10 openstackgerrit Surya Seetharaman proposed openstack/nova stable/queens: Extending delete_cell --force to delete instance_mappings https://review.openstack.org/553478
16:53:18 sahid_ cfriesen: i think you also have to look at the method where we define domain xml
16:53:25 edleafe efried: looking...
16:54:18 sahid_ cfriesen: get_config_xml or somethinf
16:54:50 openstackgerrit sahid proposed openstack/nova master: libvirt: slow live-migration to ensure network is ready https://review.openstack.org/497457
16:54:53 jaypipes artom: I don't think it's necessary to depend on bauzas' spec. I think that would endanger making progress on fixing live migration for NUMA/pinning
16:56:14 artom jaypipes, right, so leave the NUMA RPs thing to itself, and just fix the pin mappings for now. I'd tend to agree, for what it's worth - feels like a more incremental step, and not a major overhaul
16:57:20 jaypipes artom: ++
16:57:28 cfriesen sahid_: in the _hard_reboot() case we call destroy(), which does _destroy() and then cleanup(), and that will undefine the instance. Arguably that's the point where we should remove the TCP port from ALLOCATED_PORTS. Then we generate new xml with new TCP ports and all should be happy.
16:57:49 jaypipes artom: and all I was asking was that you make a little more explicit in the spec what is and what isn't "claimed".
16:58:06 cfriesen sahid_: removing the TCP ports from ALLOCATED_PORTS while they're still defined in the domain is just asking for trouble.
16:58:33 artom jaypipes, totally fair.
16:58:37 openstackgerrit Merged openstack/nova stable/pike: doc: fix the link for the evacuate cli https://review.openstack.org/542856
16:58:58 artom jaypipes, err, you're using "claimed" because the compute node doesn't actually talk to placement to "claim" the pinned pCPUs, right?
17:01:32 sahid_ cfriesen: if you can make all of that better i will be happy to review any of the your patches
17:01:40 jaypipes artom: no, I'm saying that "claim resources" means something very specific in the scheduler -- it is the call to placement to PUT /allocations/{instance_uuid}. And that does *not* include any NUMA resources right now. So I want the spec to be clear about that. When you say "claim in the scheduler", that's not actually what happens. The "claim on the compute" is the old way of allocating resources from the compute node to the instance by writing
17:01:40 jaypipes the record to the compute_nodes cell DB table. that is still done for NUMA and PCI resources in the resource tracker's instance_claim() method.
17:02:45 sahid_ dansmith: i updated the patch related to live-migration, the point is to have if possible something like an agrement on one of the version so i could make it tested internally
17:03:41 artom jaypipes, thanks for setting me straight :)
17:03:41 dansmith sahid_: I already said the implementation looks right, barring the gaps in testing
17:04:26 cfriesen sahid_: cool, if I get some time I'll hold you to that. :)
17:05:06 cfriesen jaypipes: artom: this spec only talks about CPU pinning, but we also need to recalculate the destination NUMA node for hugepage-backed instances even without CPU pinning.
17:05:11 sahid_ dansmith: i would like avoid any difference, so if you have a moment please have a look in the last version
17:05:15 sahid_ cfriesen: :)
17:06:34 artom cfriesen, jaypipes, so should we just extend this to NUMA live migration, all the while keeping the old claim on the compute way of allocating resources?
17:07:09 dansmith sahid_: you didn't answer my question about the neutron events in the tests
17:07:29 claudiub|2 dansmith: hellou. Ehm, I saw that we don't allow certain DB operations in nova (drops and alters). I'm trying to add an item to an enum, but afaik, that requires an alter. Or is there a better way to do it?
17:08:34 dansmith claudiub|2: yeah, we banned alters because they're not (usually) additive and doable online.. I think we had one in the past we exempted because we confirmed with jaypipes that it was lightweight.. does yours fit that description?
17:08:58 cfriesen artom: yes, I think it's really NUMA-aware live migration.
17:09:27 claudiub|2 nope, I'm trying to add an item to the Migration.migration_type enum.
17:09:51 dansmith claudiub|2: oh just adding something to an existing enum?
17:09:57 claudiub|2 yep
17:10:01 dansmith jaypipes: ^ hopefully that is not a big deal to do online?
17:10:28 claudiub|2 this is the commit: https://review.openstack.org/#/c/185961/4 but postgresql seems unhappy about it.
17:10:32 jaypipes dansmith: no, it's virtually instantaneous
17:10:37 dansmith jaypipes: ack
17:10:57 dansmith claudiub|2: so I think there should be at least one more exception in whatever test that is, so you can copy that for yours I think
17:11:02 artom cfriesen, not a bad idea :)
17:11:17 jaypipes honestly, we really shouldn't be using the ENUM type anyway... but meh

Earlier   Later