Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-05-10
19:32:24 dansmith is that if I boot an instance, ssh in, check the AZ that it's in, and then boot an instance in another az to make sure I have separation,
19:32:34 dansmith then something moves my instance into the same az and I'm surprised
19:32:39 melwitt and they're stuck because it won't let the AZ be crossed
19:32:41 dansmith which won't happen now
19:33:00 dansmith melwitt: yeah but they have no idea if there is space in the cloud, or in their az
19:33:59 mriedem one fear i have is operators will rely on the force flag for live migrate or evacuate to bypass the scheduler now to move the instance into another compute just because that other compute is in another AZ,
19:34:01 mriedem but idk
19:34:02 melwitt yeah, not until they contact support I guess, and admin can see the migration should be able to work and it doesn't
19:34:38 mriedem if only we could force a cold migration to bypass the scheduler.... :)
19:34:40 dansmith mriedem: fair point, but not super strong, IMHO
19:34:46 mriedem dansmith: agree it's not strong
19:35:04 mriedem if you have 10 computes in AZ1 and 10 in AZ2, then just pick one of the ones that's already in the same AZ
19:35:38 mriedem also, kablam https://review.openstack.org/567682
19:38:19 openstackgerrit Arvind Nadendla proposed openstack/nova master: update scheduler to use image-traits https://review.openstack.org/560596
19:40:36 clutchbeyers mriedem: cold migrations bypassing the scheduler was a joke right? :)
19:42:31 mriedem clutchbeyers: L645 https://etherpad.openstack.org/p/nova-ptg-rocky
19:43:01 mriedem oh the -5 didn't make it into the notes
19:50:48 melwitt dansmith: I was just reading the ML thread about the map_instances marker thing, and I think what takashin was suggesting is to print the next marker before returning 1, and then let that be passed as a starting point when the command is run again https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1193
19:51:03 melwitt would that not work for some reason?
19:57:53 edleafe melwitt: if the batches were all done in a row, it might. But there could be long gaps of time between batches, so the marker needs to be stored
19:58:23 melwitt I see
20:07:08 openstackgerrit Arvind Nadendla proposed openstack/nova master: update scheduler to use image-traits https://review.openstack.org/560596
20:07:38 dansmith melwitt: yeah I know he was talking about that and I agree with edleafe that it's not a great solution
20:07:54 dansmith melwitt: the bit I'm not sure about was the "it could be exposed accidentally through the api" part
20:09:20 melwitt okay, I think I missed the "exposed accidentally through the api" part
20:09:51 melwitt I didn't see anyone mention something like that
20:10:04 dansmith he said "in some compute rest apis, it returns the marker"
20:10:26 melwitt I think he means like in instance list, it returns the marker if there's additional pages available
20:11:06 melwitt like if you have > 1000 (api max limit) instances in the list, it will give you a "next marker" so you can query again and pass the marker to get the next page
20:11:32 dansmith you mean you think he's offering the REST API's pagination as an example that could be used for the cli to be user-supplied-marker behavior?
20:12:03 melwitt that's what I think he was saying, yeah. if doing it similar to that could avoid storing the marker
20:12:32 dansmith I guess I can see it being read like that, but to me it sounded like he was saying it's bad to store it in-band because there's a way you can see it externally
20:12:41 melwitt if map_instances hands you back the next marker, says "I did a batch but there's more" and you can call again and give the marker and it does the next batch
20:14:37 dansmith I understand how the user could supply a marker, yeah
20:15:00 dansmith I just think that's fairly terrible UX for a CLI, especially a chatty one that has all kinds of other crap going to stdio like python warnings
20:15:39 melwitt oh, hm. yeah, I guess if I re-read it and connect the 3rd paragraph to the 2nd, it could sound that way. but I thought he was stating it's not ideal to store the marker and then next gave a suggestion on how we could avoid storing it
20:16:07 dansmith I'll send another clarification mail
20:16:38 arvindn05 efried: gibi: mriedem: updated the code review for https://review.openstack.org/560596 - "update scheduler to use image-traits"
20:16:43 melwitt yeah, maybe it doesn't fit in so well for the CLI, but it's an interesting idea. probably the only way to avoid having to store a marker record
20:17:01 efried arvindn05: ack
20:19:08 arvindn05 efried: ty...hopefully the final PS for this change :)
20:19:18 efried arvindn05: Hah!
20:19:37 efried arvindn05: You've just karmically guaranteed that won't be the case!
20:19:52 dansmith melwitt: it's what we had before we replaced it with this, because it was annoying
20:20:04 dansmith https://review.openstack.org/#/c/301916/
20:20:05 melwitt ah, okay
20:20:24 mriedem fwiw,
20:20:29 arvindn05 efried: lol...well to job security then...hehe
20:20:37 efried :)
20:20:58 mriedem the new heal_allocations CLI i added has a note in there about taking a marker from the user, and dumping out the last instance processed so the next run can start from the marker rather than process a bunch of instances already healed
20:22:09 dansmith mriedem: we have all these other instances where we're able to keep track of things for the admin, it feels pretty sucky to go back to "you must have a mouse and a scratch pad to use this thing" experience
20:22:16 dansmith discovery hosts, instance mapping, reqspec fixup
20:22:26 melwitt that's a really old change too, I somehow stayed blissfully unaware of how the map_instances CLI worked historically
20:22:35 dansmith all keep track of it for you because we're talking about potentially an ass-ton of 50-item iterations
20:23:17 mriedem i didn't realize we had a marker for the reqspec data migration
20:23:21 dansmith yup
20:23:39 dansmith it wasn't as controversial because it wasn't fighting a UC field
20:23:43 dansmith but it worked basically the same way
20:24:12 mriedem i'm not even sure where i'd store the marker in my case
20:24:22 mriedem i'm iterating the instances table per cell
20:24:30 dansmith yeah, I understand yours is a little less obvious, but these all have reasonable places
20:24:39 mriedem and we can't put a fake instance in the instaces db
20:24:42 mriedem *table
20:25:21 dansmith we could have a scratch pad table we use for this kind of thing.. just "varchar(255) key, text value"
20:25:32 dansmith no index, never expect to have much data in there,
20:25:38 dansmith just some context for things like this
20:25:54 mriedem well, i thought about system_metadata,
20:26:20 mriedem because i could query for a specific system_metadata with a marker key, get that instance, delete the marker entry and continue, but it's not great
20:26:51 dansmith you could yeah, but it could be a little racy with things potentially loading/saving that instance for other things maybe
20:26:56 dansmith although you're just doing active things, so..
20:27:22 dansmith I would like to circle back,
20:27:35 dansmith and point out that the only reason we're discussing this is because we emit a few warnings during unit tests :)
20:30:03 mriedem that's not why i'm talking about it,
20:30:06 mriedem i don't care about the map_instances thing
20:30:18 efried arvindn05: I was totally going to let the typos slide, but that last docstring is wrong enough to warrant a respin.
20:30:23 efried arvindn05: karma ^^
20:31:10 melwitt it's true that the unit test warnings drew attention to the marker-record-storing pattern. but I think lots of people have been unaware of that pattern and it's pretty surprising, or at least it was to me the first time I learned about it
20:31:34 arvindn05 arvindn05: ...good catches...will fix ASAP
20:31:38 dansmith ...because it has just been working? :)
20:32:19 openstackgerrit Matt Riedemann proposed openstack/nova master: Add contributor docs on deprecating and removing compute REST APIs https://review.openstack.org/567687
20:32:21 melwitt yep, I guess so. no problems, no eyeballs drawn to it
20:35:00 mriedem dansmith: want to hit these backports https://review.openstack.org/#/q/Idea86d6bb648b1e6fef8813dbe569724ce81a750 - then we can safely remove the legacy live migration job from project-config and openstack-zuul-jobs
20:35:07 mriedem dansmith: shouldn't require 2 +2s i don't think
20:35:20 dansmith OMG YES
20:35:26 mriedem otherwise right now we're getting hit on master b/c we're running 2 live migration jbos
20:35:26 dansmith <-- loves hitting backports
20:35:46 mriedem it will take that as (1) serious and (2) literal
20:35:58 dansmith mriedem: hah
20:36:05 melwitt :D
20:36:10 mriedem i've been holding https://review.openstack.org/#/q/Ib10081150e125961cba19cfa821bddfac4614408
20:36:34 mriedem then we can do a (sweet) release
20:36:52 mriedem this is ready too https://review.openstack.org/#/c/567271/
20:39:45 openstackgerrit Merged openstack/nova master: Remove mox in test_compute_api.py (1) https://review.openstack.org/564415
20:40:40 mriedem oomichi: ha https://review.openstack.org/#/c/564415/3/nova/tests/unit/compute/test_compute_api.py@1013
20:40:52 mriedem oomichi: check out the vmware virt driver unit tests at some point
20:41:15 oomichi mriedem: hehe, more longer ? :)
20:43:38 openstackgerrit Merged openstack/nova master: libvirt: check image type before removing snapshots in _cleanup_resize https://review.openstack.org/566367
20:44:18 mriedem ok now i can bug dan about those backports
20:45:53 melwitt nova meeting in 15 minutes
20:48:13 openstackgerrit Jay Pipes proposed openstack/nova master: placement: always create consumer records https://review.openstack.org/567678

Earlier   Later