| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-31 | |||
| 18:26:34 | sdague | mriedem: ah, bummer | |
| 18:26:59 | mriedem | sdague: the spec is about the general plan | |
| 18:27:12 | mriedem | just like the original resource providers spec was about the general plan, and changed a lot over time | |
| 18:27:28 | sdague | ok | |
| 18:28:08 | mriedem | i also have http://forumtopics.openstack.org/cfp/details/31 proposed | |
| 18:28:40 | mriedem | i agree poc would be ideal | |
| 18:29:15 | mriedem | chances are this won't happen in nova in pike anyway, since there are dependencies on changes in searchlight first | |
| 18:29:17 | mriedem | which don't have owners | |
| 18:29:40 | mriedem | but it's The Plan (tm) | |
| 18:32:27 | sdague | yeh, I guess I'd just soften it further from the recommended mode, given we have no idea if it's going to work. Just say, we think this is going to be a better search approach, and we want to implement an optional backend to get it into the field. | |
| 18:32:46 | sdague | I don't think this is nearly as straight forward an aggressive cut over as placement | |
| 18:32:57 | openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: WIP: Add the specified microversion into error msg https://review.openstack.org/451986 | |
| 18:32:58 | openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: Remove unnecessary duplicated NOTE https://review.openstack.org/452284 | |
| 18:33:25 | mriedem | i wouldn't say integrating placement has been straightforward | |
| 18:40:23 | mriedem | melwitt: so, i hate to talk about this, but before i fix the bug that's shown in https://review.openstack.org/#/c/449334/ - i'm thinking about what you said where were are committing the quota usage decrement in the api before we delete the instance, and on failure we try to rollback, but that doesn't work | |
| 18:40:52 | mriedem | i noticed something funny in the quota object | |
| 18:41:07 | melwitt | okay | |
| 18:41:10 | mriedem | on commit and rollback, if the reservation is not set, it's a noop | |
| 18:41:11 | mriedem | https://github.com/openstack/nova/blob/master/nova/objects/quotas.py#L100 | |
| 18:41:16 | mriedem | https://github.com/openstack/nova/blob/master/nova/objects/quotas.py#L111 | |
| 18:41:22 | mriedem | i was wondering why we didn't fail | |
| 18:41:38 | mriedem | but those are really invalid operations, if you're trying to commit or rollback reservations before you made the reservation, it's a bug | |
| 18:41:40 | mriedem | right? | |
| 18:41:51 | melwitt | fwiw, I think it's a no-op at the db/api level too, so I don't think it matters | |
| 18:42:06 | mriedem | i'm not sure that's a good reason though | |
| 18:42:13 | melwitt | sure | |
| 18:42:27 | mriedem | but yeah, in the db api methods, if they don't find a reservation, they don't do anything | |
| 18:42:36 | mriedem | i was wondering why we didn't get a NotFound error on delete | |
| 18:42:36 | melwitt | yeah, it's not valid to do a commit/rollback without a reserve first | |
| 18:42:55 | mriedem | https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L3934 | |
| 18:43:11 | mriedem | https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L3922 | |
| 18:43:13 | melwitt | I'm not sure what the reasoning is to do a no-op, if there's some case where that would be a thing | |
| 18:43:15 | mriedem | reservations are just uuids right? | |
| 18:43:29 | melwitt | reservations are records in the reservations tables | |
| 18:43:32 | melwitt | *table | |
| 18:43:36 | mriedem | with a uuid | |
| 18:43:36 | mriedem | yeah | |
| 18:43:40 | mriedem | filter(models.Reservation.uuid.in_(reservations)).\ | |
| 18:44:00 | mriedem | https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L3866 | |
| 18:44:03 | mriedem | is on reserve | |
| 18:44:06 | mriedem | we get the list of uuids back | |
| 18:44:09 | mriedem | and store that in the Quota object | |
| 18:44:20 | artom | Can an instance have multiple fixed IPs? | |
| 18:44:36 | mriedem | so maybe i'll tinker with making commit/rollback w/o a reservation in the object a failure | |
| 18:44:42 | melwitt | artom: I think so, if it's on multiple networks | |
| 18:44:43 | mriedem | and see what falls out | |
| 18:44:49 | artom | melwitt, thanks :) | |
| 18:44:51 | mriedem | artom: yeah definitely | |
| 18:44:53 | clarkb | mriedem: artom also ipv4 + ipv6 | |
| 18:44:54 | mriedem | multinic | |
| 18:45:03 | clarkb | er melwitt artom ^ | |
| 18:45:08 | mriedem | and ipv8 | |
| 18:45:11 | mriedem | it's secret | |
| 18:45:12 | mriedem | and new | |
| 18:45:19 | mriedem | full of tomatoe | |
| 18:45:27 | artom | Are we running out of quarks to give IPv6s to? | |
| 18:45:38 | artom | IoSAP? | |
| 18:45:47 | artom | Internet of SubAtomic Particles... | |
| 18:47:05 | artom | Eesh, one bad joke and everyone shuts up? | |
| 18:47:52 | artom | Wear a bib next time ;) | |
| 18:48:05 | mriedem | you'd have to warn me | |
| 18:48:12 | mriedem | unless i start wearing one full time | |
| 18:48:49 | melwitt | mriedem: yeah, might be worth it to do an experiment. I can't imagine why no-op would be valid other than some sort of race where commit could possibly hit before the reserve and that leaving an orphan reservation is okay | |
| 18:49:12 | melwitt | I can't think of such a race, but I have heard of having a lot of orphaned reservations in the db. so that implies it happens somehow | |
| 18:49:16 | mriedem | maybe comstud is lurking | |
| 18:51:44 | mriedem | interesting https://github.com/openstack/nova/blob/master/nova/quota.py#L1401 | |
| 18:52:09 | mriedem | same with rollback https://github.com/openstack/nova/blob/master/nova/quota.py#L1425 | |
| 18:52:11 | mriedem | we just ignore exceptions | |
| 18:52:13 | mriedem | any exceptions | |
| 18:52:25 | mriedem | no matter how much Vek says that's safe, it seems unsafe | |
| 18:52:40 | melwitt | heh | |
| 18:52:48 | mriedem | " because the usage resynchronization will resolve the issue" | |
| 18:52:53 | mriedem | we don't re-sync by default do we? | |
| 18:53:00 | melwitt | we do. on every reserve | |
| 18:53:39 | melwitt | it will do a sync for that project/user combo | |
| 18:53:56 | melwitt | it doesn't help if some other user in your project is out of sync though | |
| 18:54:34 | mriedem | https://github.com/openstack/nova/blob/master/nova/conf/quota.py#L278 | |
| 18:54:37 | mriedem | i was thinking of ^ | |
| 18:54:40 | melwitt | also a reminder that quota counting will remove reservations and usages | |
| 18:54:56 | melwitt | oh, hm | |
| 18:54:59 | mriedem | and https://github.com/openstack/nova/blob/master/nova/conf/quota.py#L267 | |
| 18:55:05 | mriedem | so i think by default, we don't refresh any of that stuff | |
| 18:55:07 | mriedem | or resync | |
| 18:55:19 | mriedem | so maybe the comment from Vek was more about, it will fix it, because we at rackspace turn those on... | |
| 18:55:19 | melwitt | yeah, okay. I didn't realize that defaulted to off | |
| 18:55:38 | melwitt | if it's on, then on every reserve it does a refresh if needed | |
| 18:55:48 | mriedem | https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L3647 | |
| 18:55:50 | mriedem | yeah if needed | |
| 18:56:00 | mriedem | based on age and until_refresh count | |
| 18:56:05 | melwitt | I think most operators turn it on, since going out of sync happens a fair bit | |
| 18:56:45 | mriedem | so if blindly handling all exceptions https://github.com/openstack/nova/blob/master/nova/quota.py#L1425 and assuming we're ok is dependent on those config options, which default to not doing the thing that comment says we do automatically, | |
| 18:56:52 | mriedem | it seems we should reraise the exception if those options aren't set | |
| 18:56:55 | mriedem | at least one of them | |
| 18:57:02 | mriedem | otherwise we're just silently failing | |
| 18:57:05 | melwitt | I guess I mean if usage != resource count if refreshes. not that that saves a lot I guess | |
| 18:57:55 | melwitt | maybe. it's going to be moot soon, we hope | |
| 19:01:57 | mriedem | what kind of state is that series in now? | |
| 19:03:25 | melwitt | I'm working on it and plan to push the series up today and see what tempest thinks of it | |
| 19:04:07 | melwitt | there is light at the end of the tunnel finally | |
| 19:12:00 | mriedem | ok so by default, reservations expire after 24 hours, | |