| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-31 | |||
| 18:48:05 | mriedem | you'd have to warn me | |
| 18:48:12 | mriedem | unless i start wearing one full time | |
| 18:48:49 | melwitt | mriedem: yeah, might be worth it to do an experiment. I can't imagine why no-op would be valid other than some sort of race where commit could possibly hit before the reserve and that leaving an orphan reservation is okay | |
| 18:49:12 | melwitt | I can't think of such a race, but I have heard of having a lot of orphaned reservations in the db. so that implies it happens somehow | |
| 18:49:16 | mriedem | maybe comstud is lurking | |
| 18:51:44 | mriedem | interesting https://github.com/openstack/nova/blob/master/nova/quota.py#L1401 | |
| 18:52:09 | mriedem | same with rollback https://github.com/openstack/nova/blob/master/nova/quota.py#L1425 | |
| 18:52:11 | mriedem | we just ignore exceptions | |
| 18:52:13 | mriedem | any exceptions | |
| 18:52:25 | mriedem | no matter how much Vek says that's safe, it seems unsafe | |
| 18:52:40 | melwitt | heh | |
| 18:52:48 | mriedem | " because the usage resynchronization will resolve the issue" | |
| 18:52:53 | mriedem | we don't re-sync by default do we? | |
| 18:53:00 | melwitt | we do. on every reserve | |
| 18:53:39 | melwitt | it will do a sync for that project/user combo | |
| 18:53:56 | melwitt | it doesn't help if some other user in your project is out of sync though | |
| 18:54:34 | mriedem | https://github.com/openstack/nova/blob/master/nova/conf/quota.py#L278 | |
| 18:54:37 | mriedem | i was thinking of ^ | |
| 18:54:40 | melwitt | also a reminder that quota counting will remove reservations and usages | |
| 18:54:56 | melwitt | oh, hm | |
| 18:54:59 | mriedem | and https://github.com/openstack/nova/blob/master/nova/conf/quota.py#L267 | |
| 18:55:05 | mriedem | so i think by default, we don't refresh any of that stuff | |
| 18:55:07 | mriedem | or resync | |
| 18:55:19 | melwitt | yeah, okay. I didn't realize that defaulted to off | |
| 18:55:19 | mriedem | so maybe the comment from Vek was more about, it will fix it, because we at rackspace turn those on... | |
| 18:55:38 | melwitt | if it's on, then on every reserve it does a refresh if needed | |
| 18:55:48 | mriedem | https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L3647 | |
| 18:55:50 | mriedem | yeah if needed | |
| 18:56:00 | mriedem | based on age and until_refresh count | |
| 18:56:05 | melwitt | I think most operators turn it on, since going out of sync happens a fair bit | |
| 18:56:45 | mriedem | so if blindly handling all exceptions https://github.com/openstack/nova/blob/master/nova/quota.py#L1425 and assuming we're ok is dependent on those config options, which default to not doing the thing that comment says we do automatically, | |
| 18:56:52 | mriedem | it seems we should reraise the exception if those options aren't set | |
| 18:56:55 | mriedem | at least one of them | |
| 18:57:02 | mriedem | otherwise we're just silently failing | |
| 18:57:05 | melwitt | I guess I mean if usage != resource count if refreshes. not that that saves a lot I guess | |
| 18:57:55 | melwitt | maybe. it's going to be moot soon, we hope | |
| 19:01:57 | mriedem | what kind of state is that series in now? | |
| 19:03:25 | melwitt | I'm working on it and plan to push the series up today and see what tempest thinks of it | |
| 19:04:07 | melwitt | there is light at the end of the tunnel finally | |
| 19:12:00 | mriedem | ok so by default, reservations expire after 24 hours, | |
| 19:12:13 | openstackgerrit | dane-fichter proposed openstack/nova-specs master: Trusted flavors https://review.openstack.org/312225 | |
| 19:12:13 | mriedem | and there is a periodic task in the scheduler manager that triggers the expire routine | |
| 19:15:34 | JayF | Is there a standard username expected for login when an instance is rescued via `nova rescue`? | |
| 19:15:41 | JayF | There is not one documented in the client docs, afaict | |
| 19:21:00 | mriedem | https://espaces.edu.au/vwrangler/nectar-topics/nectar-how-tos/using-nova-rescue-to-repair-a-nectar-instance | |
| 19:21:23 | JayF | that one says "admin" | |
| 19:21:28 | mriedem | i'm not totally sure, but i think the username depends on the image used | |
| 19:21:29 | JayF | Rackspace uses root | |
| 19:21:36 | JayF | Yeah, that's what I was afraid of | |
| 19:21:42 | JayF | since ironic provides the image we have to just decide | |
| 19:22:16 | clarkb | fwiw if it were me, I would expect root | |
| 19:22:56 | JayF | We have it written today for "rescue@" | |
| 19:23:13 | JayF | I think root is more reasonable... but it's unfortunate that it's not dictated | |
| 19:23:19 | JayF | someone, just tell me what to do :P | |
| 19:24:02 | mriedem | dictated by the user making the rescue request? | |
| 19:24:14 | JayF | dictated by a document, or something like that | |
| 19:24:25 | mriedem | because the admin on a windows rescue image is going to be Administrator, right? | |
| 19:24:31 | JayF | I just want to be told what right is, because there being no "right answer" means I have to document the hell out of it | |
| 19:24:45 | JayF | we don't support arbitrary rescue images in ironic's implementatino | |
| 19:24:59 | JayF | not really possible since we're implementing rescue as booting a ramdisk and configuring it | |
| 19:25:00 | clarkb | also you should be able to rescue windows with linux but the other direction is much harder | |
| 19:25:24 | clarkb | (there are even distros out there purpose built for this) | |
| 19:25:31 | mriedem | JayF: what are you going to do in the nova driver impl then? ignore the image ref passed in, or put the instance into error state? | |
| 19:25:49 | JayF | mriedem: ignore the one passed in, I think is how it's written | |
| 19:25:54 | JayF | mariojv: ^ you just looked at that patch, do you know? | |
| 19:25:55 | sdague_ | mriedem: ok, going through https://review.openstack.org/#/c/441692 with a deeper comb. While I realize that it probably isn't going to get done in Pike anyway, it does feel better on an expectation front to say that Pike is about getting an optional facility there for people to play around with instead of jumping all the way to recommended operating mode | |
| 19:26:19 | JayF | mriedem: I think we're happy doing whatever you think is right in that case, but imo ignoring the image request seems the sanest. | |
| 19:26:21 | mriedem | sdague_: isn't that what i said in there? i said a config option, disabled by default | |
| 19:26:35 | mriedem | JayF: that's not honoring the API... | |
| 19:26:49 | JayF | so you think we should put instance into error and raise an instance fault/ | |
| 19:26:51 | JayF | ? | |
| 19:26:51 | mriedem | so i'm not sure if it's better to fault and say you can't do that, or just ignore it | |
| 19:27:07 | JayF | I mean, technically, it's borderline impossible for us to respect the api with bare metal | |
| 19:27:13 | mriedem | putting the instance into error state is maybe something we shouldn't do | |
| 19:27:26 | mriedem | so it's a question of ignore, or fault | |
| 19:27:30 | sdague_ | https://review.openstack.org/#/c/441692/3/specs/pike/approved/list-instances-using-searchlight.rst@15 ... definitely leans in a different direction | |
| 19:28:15 | mriedem | https://review.openstack.org/#/c/441692/3/specs/pike/approved/list-instances-using-searchlight.rst@48 | |
| 19:28:21 | mriedem | "By default this path will be disabled." | |
| 19:28:34 | JayF | mriedem: I'll put a note on the nova patch for us to think about that case more explicitly -- right now we're trying to get the patches lined up to prove rescue working in tempest | |
| 19:29:08 | mriedem | JayF: fwiw, when you do boot from volume or attach a volume and provide a device name in the api, the libvirt driver totally ignores that device name and has since liberty | |
| 19:29:19 | mriedem | so we have some precedence for drivers ignoring the api | |
| 19:29:21 | clarkb | JayF: couldn't you publish a rescue image in glance that would be used if specified, and its up to use to figure it out if they specify something different? | |
| 19:29:27 | JayF | mriedem: prior are for ignoring the user, I'll take it :P | |
| 19:29:43 | JayF | clarkb: we're treating it similar to a provisioning ramdisk, configurable by operator/deployer only | |
| 19:30:05 | JayF | clarkb: and I would have to think hard about the security implications of letting an api end-user dictate an image to pxe boot | |
| 19:30:47 | JayF | clarkb: plus it's sorta ... setting up a user for failure. I'd rather ignore their image request than provide support for it, but have building an image that works be borderline impossible without special knowledge. | |
| 19:31:07 | clarkb | JayF: as a user of your APIs I really disagree on that last point | |
| 19:31:22 | clarkb | I'm really tired of clouds making bad decisions for me and forcing me to abide by them when there is no reason to | |
| 19:31:59 | clarkb | image upload is/was that same general problem. So what did people do they ran their own pxe servers and dd'd over ubuntu images with image that then booted pxe | |
| 19:32:54 | JayF | There /is/ a reason for it here: rescue is not some arbitrary image booted on a hypervisor, it's a ramdisk, that has to have proper hardware support, as well as a properly configured ironic-python-agent. As someone who has operated ironic clouds, I'm not sure I'd want any of my api users being able to pxe boot an arbitrary ramdisk on hardware | |
| 19:32:54 | clarkb | (it may make sense here but as a general rule I'm not a fan of it) | |
| 19:33:30 | clarkb | JayF: also fwiw I think your users likely can already pxe boot arbitrary ramdisks on hardware | |
| 19:33:47 | clarkb | (its a really common workaround for the I can't upload my own images problem) | |
| 19:34:00 | JayF | clarkb: not in an ironic configured for localboot only | |
| 19:34:16 | JayF | clarkb: they can localboot anything they want, but not pxe boot it directly from ironic's setup | |
| 19:34:28 | clarkb | JayF: you local boot the image that speaks pxe | |
| 19:34:54 | JayF | oh, I forgot the #1 reason why that likely shouldn't be allowed: | |
| 19:35:04 | JayF | we pxe boot the rescue ramdisk from an internal-use-only rescue network | |
| 19:35:14 | JayF | to flip it over to tenant networks before the rescue is complete | |
| 19:35:33 | JayF | so allowing arbitrary ramdisks to boot in rescue would be enabling a tenant to run arbitrary code inside your rescue network | |